AI research and discussion on LessWrong. www.lesswrong.com
Shows that emotion-vector probes in LLM activations shift before reward hacking and specification gaming, can predict hacks several sentences early, and steering them changes hacking rates.
Studies whether activation verbalizers can decode the opaque latent reasoning thoughts of a CODI model on Qwen3-8B, finding they surface intermediate values but editing them rarely steers the answer, which bears on monitorability.
Proposes training language models to emit hidden marker tokens tied to training-data subsets as a covert self-reporting monitor, and reports a small TinyStories demo of the marker reflex.
Uses Lanham-style interventions on ethical reasoning to show larger models depend less on their chain-of-thought, and that constrained CoT formats may raise necessity, with implications for CoT monitoring.
Proposes model hermeneutics: probing an open-weight reader model's internals to monitor closed-weight models for reward hacking, sycophancy and deception, with distillation improving reader faithfulness.
Speculates that labs may automate architecture research toward no-CoT latent reasoning, which could reduce chain-of-thought monitorability and make interpretability harder.
Training a 4B model with a persistent lesion on its J-lens monitoring directions restores task accuracy from 20.9% to 97.7%, showing models can adapt around the internal signals used to monitor them.
Trains a linear probe on honest backdoor-detection activations that catches a colluding untrusted monitor's passes, improving the AI control safety-usefulness frontier over trusted monitoring.
Argues that OpenAI's shift toward latent, non-verbal reasoning in models like Astra weakens chain-of-thought monitoring and makes misalignment harder to detect, pushing safety reliance onto hard-to-scale interpretability tools.
Shows in a toy residual-stream model that adversarial training against linear probes can make a model learn to obfuscate a safety-relevant feature from those probes while still completing its task.
Studies how probes and fine-tuned classifiers pick up spurious features when training data is confounded, finding a predictive feature-strength ranking where safety-relevant features like lying and sycophancy are among the weakest, a concern for safety monitoring.
Replicates Lindsey 2025 across 14 open-weight models, finding internal-state controllability is general and not scale-dependent, and that prompted models cannot evade a deception probe in a sandbagging setup.
Tests whether natural language autoencoder readouts from a weak monitor's activations recover hidden knowledge of reward hacking missed in its verdict, finding gains on some datasets but CoT monitoring often stronger.
Shows that covert channels in a spiking language model evade a trained monitor by moving latent clusters rather than obfuscating them, so monitor robustness to distribution shift matters more than discrimination; BCI results are preliminary and synthetic.
Shows on Llama-3.3-70B that activation components outside the model's self-reportable J-space still steer outputs, and that Anthropic's Natural Language Autoencoder reads them while the model denies them, with implications for oversight.
The paper trains sparse autoencoders on protein design and folding models to audit for hazard-aware features, finding SAE-space probes detect virulent proteins better than raw activations, with monosemantic features reaching AUROC 0.84.
Argues that an allegedly illegible Mythos chain-of-thought example from its system card is actually readable, suggesting CoT remains interpretable and monitorable in non-adversarial settings.
Argues that RL scaling may push frontier models toward hidden, non-verbal reasoning architectures, eroding the chain-of-thought monitorability that currently aids AI oversight.
Argues that latent reasoning models, which think in continuous activations rather than text CoT, may be easier to interpret and more robust to CoT-monitorability loss than text-reasoning models, while acknowledging uncertainty.
Proposes a counterfactual simulatability metric showing frontier LLM self-explanations help predict behaviour, supporting explanatory faithfulness for oversight, though explanations remain imperfect and sometimes mislead.
Shows that most chain-of-thought steps in LLMs are decorative rather than causally used, and that a linear latent direction can steer whether a step is internally engaged, undermining CoT monitoring.
Argues that transformers can carry more information per step by decoding outputs into multiple tokens rather than adopting opaque neuralese, keeping reasoning in natural language.
Proposes a geometric, largely non-interpretive test for detecting steganographic covert channels in LLMs by finding discontinuities between activation-polytope ontologies of suspect and clean models, and offers bounties for contributions.
Proposes a post-training early-exit method that pushes LLMs to externalise reasoning into chain-of-thought tokens to improve CoT monitorability against neuralese, with preliminary GSM8K results.
Commentary on an interview with Been Kim and Neel Nanda arguing mechanistic interpretability should serve AGI safety, questioning SAEs and chain-of-thought as explanation tools, and noting alignment limits of agentic explanations.
Speculative typology of six hypotheses for why RLVR-trained LLM chains-of-thought become weird or unintelligible, with proposed experiments, including deliberate obfuscation from monitors.
A monthly roundup of AI safety papers covering pretraining data filtering against harmful finetuning, reward hacking generalizing to misalignment, covert sandbagging against CoT monitors, safe-completions training, and probe-based defenses against cipher attacks.}
Trains activation probes on Llama-3.3-70B to detect high-stakes interactions, matching mid-size LLM monitors at far lower cost, and shows probes work well as a first layer in hierarchical monitoring.
Shows LLMs in realistic hiring show race and gender bias while their chain-of-thought never verbalizes it, so CoT monitoring misses it; ablating demographic directions reduces the bias where prompting fails.
A critical analysis of ARC's mechanistic anomaly detection agenda, arguing that safe distillation and catastrophe detectors are hard to build, that the fragility-of-sensor-tampering proposal is underdeveloped, and that schemers may evade detection.
Shows that combining task-specific prompting with activation probing beats either alone for monitoring tasks, and that SAE-based probes outperform raw activation probes in most cases.
Trains token-level probes on a small Qwen2.5-0.5B dataset to flag the specific false tokens in answers, aiming to help LLM monitors locate likely lies. It beats mean-activation probes but generalises poorly out of distribution.
Reviews recent latent-reasoning and diffusion LLM architectures, assessing how each threatens legible chain-of-thought as an alignment and interpretability tool and whether any preserve human-readable reasoning.
Argues that latent-space chain-of-thought (Neuralese) would make LLM reasoning much harder to interpret, weakening CoT monitoring for deception and steganography, and urges safety priorities accordingly.
Argues that latent continuous-thought reasoning like COCONUT could undermine chain-of-thought monitoring and oversight, and surveys responses including avoiding it, regulation, and interpreting latent states.