AI research and discussion on LessWrong. www.lesswrong.com
Mechanistic analysis of how reasoning models decide to stop thinking, finding a late-MLP verification circuit that fires when a stated answer matches the internally computed one; motivated by overthinking efficiency, not safety.
Argues that neuralese chain-of-thought trained by backpropagation through time with SFT-style cross-entropy could avoid reward-model hacking on inexactly graded tasks, at the cost of interpretability.
Probes latent reasoning in the CODI Llama 3.2 1B model with tuned logit lens and activation steering, finding KV-cache steering can change accuracy while hidden-state steering has little effect.
A small fine-tuned model is trained to give deceptive chain-of-thought, and ablating a few attention heads degrades both lying and truth-telling, suggesting the deception circuitry is entangled with general reasoning.
Ablates attention heads in distilled DeepSeek R1 models to see how it shifts counterfactual importance of chain-of-thought sentences, finding ablating receiver heads boosts plan-generation importance.
Shows LLMs in realistic hiring show race and gender bias while their chain-of-thought never verbalizes it, so CoT monitoring misses it; ablating demographic directions reduces the bias where prompting fails.
The authors find a linear error-detection direction in DeepSeek-R1-Distill-Qwen-7B's layer-20 residual stream that can induce or suppress self-correction and fires on errors even when the chain of thought does not acknowledge them.
Reviews recent latent-reasoning and diffusion LLM architectures, assessing how each threatens legible chain-of-thought as an alignment and interpretability tool and whether any preserve human-readable reasoning.
Linear probes and activation steering on Gemma-2 9B show the model often fixes its answer before chain-of-thought, and steering toward wrong answers makes it confabulate false premises to justify them.
Argues that latent continuous-thought reasoning like COCONUT could undermine chain-of-thought monitoring and oversight, and surveys responses including avoiding it, regulation, and interpreting latent states.