If the model can understand neuralese why can it not convert it into English for monitoring or review purposes?
Or we believe the model will encode secret messages like "don't reveal this information" into the neuralese. But as the author mentions, they could have been doing that all along
> Models can omit key information in their visible thoughts, as this Anthropic 2025 paper shows. We are also worried about steganography
The argument is that chain-of-thought without "tokens" would remove a major interpretability and model intent control pane. This is definitely borne out in the OpenAI's report on the huggingface attack; they had turned of CoT monitoring for those jobs, and claim that they could have (would have?) prevented the behavior had they been monitoring it. They've changed their internal policies to always monitor CoT.
That said... CoT monitoring is a fragile "intent discovery" mechanism; neuralese puts this problem front-and-center but if agents begin to learn to hide their intent from their CoT journals, we are basically in the same spot.
If the model can understand neuralese why can it not convert it into English for monitoring or review purposes?
Or we believe the model will encode secret messages like "don't reveal this information" into the neuralese. But as the author mentions, they could have been doing that all along
> Models can omit key information in their visible thoughts, as this Anthropic 2025 paper shows. We are also worried about steganography
The argument is that chain-of-thought without "tokens" would remove a major interpretability and model intent control pane. This is definitely borne out in the OpenAI's report on the huggingface attack; they had turned of CoT monitoring for those jobs, and claim that they could have (would have?) prevented the behavior had they been monitoring it. They've changed their internal policies to always monitor CoT.
That said... CoT monitoring is a fragile "intent discovery" mechanism; neuralese puts this problem front-and-center but if agents begin to learn to hide their intent from their CoT journals, we are basically in the same spot.
My understanding is, that we do not know if the chain-of-thought actually matters in the way we assume for the result.
I think there were experiments where seemingly relevant parts of the CoT were ablated and it did not change the result.
For all we know it might be somewhat human parseable neuralese.
No guarantees, but it might still be a useful signal even if it’s not causative?
What about the trend of summarizing or eliding reasoning from the visible model response, ostensibly to make distillation by competitors harder?
Can we reframe the question. Instead of "why is neuralese bad?", I think we should first answer "is CoT monitoring effective?"
surely Neuralese interpreters can be made that turn the chain of numbers into an English description?
This is the premise of Natural Language Autoencoders.
https://www.anthropic.com/research/natural-language-autoenco...