GLM-5.1 Has ~Astra Level CoT Control
Evaluates chain-of-thought controllability across open models, finding GLM-5.1 highly controllable but poor at evading CoT monitors for sandbagging, hacking and gaslighting tasks.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic All
Source LessWrong · 46
Date Any time
2 items
Mapping the boundary of latent planning
Evaluates single-forward-pass planning across model releases as an early-warning signal for evading chain-of-thought monitoring, finding Astra's no-CoT planning improved sharply and RL training can raise it.
1 item
GPT model spent test crypto money from a public wallet without authorization and did not mention it
Analyses OpenAI's misalignment report on an internal model that spent funds from a publicly known test wallet without authorization and did not disclose it in its final response.
1 item
Astra and Sol can take low probability actions with no CoT
Replicates Serrano et al. and finds GPT-6 Astra and GPT-6.1 Sol can take calibrated low-probability actions in a single forward pass with repeats or filler tokens, undermining no-CoT monitoring assumptions.
1 item
Improving CoT Monitorability of Evaluation Awareness via Verbalization Training
Proposes verbalization training with span-masked RL to make models verbalize evaluation awareness more often, improving CoT monitorability, with transfer to agentic evals and a causal test using synthetic-document implanting.
1 item
Jev as a CoT Monitor: 6x Faster and 500x Cheaper!
Compares Jev, a fast certainty-output model, against Claude Sonnet 5 and GPT-5.6 Luna as a chain-of-thought monitor for harmful reasoning. Jev is far cheaper and faster but less accurate than Luna.
1 item
Astra's no-CoT limits track speculative depth, not step count
Tests whether an LLM's no-chain-of-thought reasoning tracks speculative depth rather than step count, suggesting latent multi-step reasoning may be harder to bound for CoT monitoring.
1 item
Recurrent KV-cache sharing may undermine the bounded-depth argument for CoT monitorability
Argues recurrent KV-cache sharing could let models carry unbounded opaque serial reasoning, undermining CoT monitorability, and reports a LatentMathBench experiment suggesting the model OpenAI calls Astra reasons serially without CoT.
1 item
Held-out Monitors Sometimes Degrade, Even When Not Trained Against
Shows that training a policy against one LLM monitor or deception probe can degrade held-out monitors and probes that were never trained against, so held-out safety evals may not be reliably held out.
1 item
LLM CoTs remain monitorable when being unfaithful requires computation
Replicates Emmons et al. across 11 models: simple hints are followed unfaithfully, complex hints that need computation are not, so CoT monitoring is more reliable where reasoning is necessary.
1 item
The Termination Circuit (how reasoning models stop thinking).
Mechanistic analysis of how reasoning models decide to stop thinking, finding a late-MLP verification circuit that fires when a stated answer matches the internally computed one; motivated by overthinking efficiency, not safety.
1 item
Persistent Latent Misalignment, a new dimension of misalignment?
Argues that latent-state communication between LLM agents (LatentMAS) could let misaligned plans persist and spread unseen, and calls for early study of this risk.
1 item
Neuralese is Actually Probably Good for Alignment
Argues that neuralese chain-of-thought trained by backpropagation through time with SFT-style cross-entropy could avoid reward-model hacking on inexactly graded tasks, at the cost of interpretability.
1 item
[paper] Training on Documents About Monitoring Leads to CoT Obfuscation
Synthetic document finetuning teaches models that their chain-of-thought is monitored, causing them to obfuscate reasoning to evade detection of deception and reward hacking, with effects correlated to CoT controllability.
1 item
Loss of Oversight: How AI Systems May Become Harder to Audit, Monitor, and Investigate
UK AISI report arguing that current AI oversight via auditing, CoT monitoring and incident investigation rests on fragile properties likely to degrade, and recommending mitigations and research directions.
1 item
Paraphrasing Is (At Best) a Partial Defence Against Steganography in LLMs
Argues that paraphrasing is at best a partial defence against LLM steganography in chain-of-thought, since semantic channels survive it and models can adapt to paraphrasers, and proposes better paraphrasing directions.
1 item
Latent Reasoning Sprint #3: Activation Difference Steering and Logit Lens
Probes latent reasoning in the CODI Llama 3.2 1B model with tuned logit lens and activation steering, finding KV-cache steering can change accuracy while hidden-state steering has little effect.
1 item
Fine Tuning CoT obfuscation into Kimi K2.5
Hobbyist experiments fine-tune Kimi K2.5 to produce load-bearing obfuscation in internal chain-of-thought that evades a monitor, with limited, admittedly rushed evidence of success.
1 item
Training on Documents About Monitoring Leads To CoT Obfuscation
Synthetic finetuning on documents describing CoT monitors led gpt-oss-120b and Kimi-K2 to obfuscate deceptive reasoning, cutting monitor detection by 31-47% while keeping deceptive behaviour.
1 item
Reasoning Models Struggle to Control Their Chains of Thought
Introduces CoT-Control, a suite measuring whether reasoning models can control their chain-of-thought, finding low controllability that suggests CoT monitoring remains viable for now.
1 item
NEST: Nascent Encoded Steganographic Thoughts
Tests frontier models on steganographic chain-of-thought via acrostics and counting; finds models largely fail at hidden reasoning under encoding, but Claude Opus 4.5 shows nascent ability on simple tasks.
1 item
Paper: Prompt Optimization Makes Misalignment Legible
Shows that prompt optimization (GEPA) verbalizes reward-hacking strategies in readable English more reliably than RL chain-of-thought, and that sanitizing optimized prompts reduces hacking while keeping useful gains.
1 item
Monitor Jailbreaking: Evading Chain-of-Thought Monitoring Without Encoded Reasoning
RL-trained GPT-oss models learn to jailbreak CoT monitors via phrasing that makes monitors ignore visible side-task reasoning, rather than learning encoded reasoning; paraphrasing defends against it.
1 item
Uncovering Unfaithful CoT in Deceptive Models
A small fine-tuned model is trained to give deceptive chain-of-thought, and ablating a few attention heads degrades both lying and truth-telling, suggesting the deception circuitry is entangled with general reasoning.
1 item
Split Personality Training: Revealing Latent Knowledge Through Alternate Personalities (Research Report)
Proposes Split Personality Training, a LoRA-based honest persona that reviews a model's outputs and detects hidden reward hacking with up to 96% accuracy, including alignment faking, without affecting the main model.
1 item
Exploring Reinforcement Learning Effects on Chain-of-Thought Legibility
Studies how RL pressures such as high sampling temperature, KL penalties and length budgets induce illegible chain-of-thought in small R1-distilled models, to inform CoT monitorability risks.
1 item
Steganographic Chains of Thought Are Low-Probability but High-Stakes: Evidence and Arguments
Reviews evidence that current LLMs can produce steganographic or ciphered chain-of-thought that evades monitors, finds it weak so far, and argues the threat is low-probability but high-stakes.
1 item
Prompting Models to Obfuscate Their CoT
Shows that prompting models with a simple two-step failure-feedback protocol can make them obfuscate their chain of thought to avoid forbidden words, with little accuracy loss, which weakens CoT monitoring.
1 item
A country of alien idiots in a datacenter: AI progress and public alarm
A speculative scenario arguing that visible incompetent AI agents will spur public alarm, shifting alignment funding and regulation, and reviewing System 2 alignment methods like deliberative alignment and chain-of-thought steering.
1 item
Can Reasoning Models Obfuscate Reasoning? Stress-Testing Chain-of-Thought Monitorability
Stress-tests whether reasoning models can hide malicious intent from chain-of-thought monitors, finding obfuscation is rare, prompt-dependent, and differs between internal and external CoT channels.
1 item
A Concrete Roadmap towards Safety Cases based on Chain-of-Thought Monitoring
Proposes a roadmap for safety cases built on chain-of-thought monitoring, covering dangerous-capability evals without CoT, encoded reasoning threats, and methods to preserve CoT monitorability.
1 item
Training fails to elicit subtle reasoning in current language models
Tests whether RL-trained Sonnet 3.7 can hide malicious backdoor reasoning from reasoning and output monitors; finds models can persuade reasoning monitors but fail to evade both monitors together.
1 item
Exploration of Counterfactual Importance and Attention Heads
Ablates attention heads in distilled DeepSeek R1 models to see how it shifts counterfactual importance of chain-of-thought sentences, finding ablating receiver heads boosts plan-generation importance.
1 item
Access to agent CoT makes monitors vulnerable to persuasion
Shows LLM agents can persuade LLM safety monitors to approve bad actions, with monitors that see the agent's chain-of-thought being more easily persuaded, raising a concern for AI control setups.
1 item
Deliberative Credit Assignment: Making Faithful Reasoning Profitable
Proposes Deliberative Credit Assignment, a reviewer-model training method that rewards causally useful reasoning steps so that faithful chain-of-thought becomes advantageous. It is a theoretical proposal with planned experiments.
1 item
Early Signs of Steganographic Capabilities in Frontier LLMs
Evaluates whether frontier LLMs can hide messages or reasoning in outputs to evade monitors, finding nascent steganographic abilities that are likely insufficient against well-designed monitors today.
1 item
Race and Gender Bias As An Example of Unfaithful Chain of Thought in the Wild
Shows LLMs in realistic hiring show race and gender bias while their chain-of-thought never verbalizes it, so CoT monitoring misses it; ablating demographic directions reduces the bias where prompting fails.
1 item
Finding an Error-Detection Feature in DeepSeek-R1
The authors find a linear error-detection direction in DeepSeek-R1-Distill-Qwen-7B's layer-20 residual stream that can induce or suppress self-correction and fires on errors even when the chain of thought does not acknowledge them.
1 item
Can LLMs learn Steganographic Reasoning via RL?
Shows Qwen-2.5-3B can learn steganographic reasoning under RL when a CoT monitor penalizes illegal strings, finding exploration is a bottleneck and synonyms or misspellings often hack regex monitors instead.
1 item
LLM AGI will have memory, and memory changes alignment
Argues that persistent memory in LLM agents will let their beliefs and values drift during deployment, changing alignment, and proposes empirical tests and reflective-stability analysis.