Research / Sources
Discussion and research on AI alignment. alignmentforum.org
Argues that latent reasoning architectures like COCONUT and full-bandwidth transformers would undermine chain-of-thought monitorability, weakening a key oversight tool against misalignment and scheming.
Shows that CoT controllability evals are heavily under-elicited: prompt optimization by Claude raises open-source models' compliance with CoT formatting constraints about 2-3x, undermining current low scores used to argue models cannot obfuscate CoTs for monitoring.
Argues that rapid military integration of AI, including classified deployment with little oversight, lowers the capabilities a misaligned AI needs for takeover, and recommends visibility and governance measures.
Proposes a conceptual framework predicting when RL training with CoT rewards makes chain-of-thought unmonitorable, classifying reward pairs as in-conflict, orthogonal, or aligned, and validates it empirically.
An opinionated essay estimating how hard AI alignment is by comparing person-years of effort against steam engine safety, Apollo, and P vs NP, and arguing for accelerating the field's growth.
Argues that improving LLM metacognition, such as error-catching and self-critique, could reduce sycophancy and slop, aiding alignment research and stability while also boosting capabilities, citing evidence that current reasoning models lag humans.
Uses logit lens, activation patching and probing to show a latent-reasoning math model stores intermediate results in specific latent vectors, motivated by the difficulty of monitoring neuralese reasoning for safety.
Evaluates 14 models and finds many reasoning models produce illegible chains of thought that still carry useful information, which threatens CoT monitoring, though legibility does not predict accuracy.
Shows that training against an output-only monitor can make chains of thought obfuscated, and proposes two mitigations (mind and face, reward targeting) that partly restore CoT monitorability across three environments.
Tests whether DeepSeek R1 and GPT OSS 120B detect tampering with their chain-of-thought; they rarely notice syntactic edits but more often notice semantic ones, and sandbagging prefills cause false alarms.
Argues that LLM-based AGI reasoning about its top-level goals could drift from trained alignment via out-of-distribution misgeneralization, and proposes empirical tests on current models.
Proposes a shared taxonomy of hidden reasoning in LLMs, including neuralese, steganography, linguistic drift and dazzling, that undermines chain-of-thought monitoring, with observed examples.
Shows that training only on honest outputs can still increase reward hacking when the training prompt omits the hack-encouraging system prompt, via reasoning-trace effects (re-contextualization), in GPT-4o-mini code tasks.
METR replicates Anthropic's CoT faithfulness setup with free-response math and harder clues, finding models are almost always faithful when CoT is needed and that a CoT-based detector reliably spots clue use.
Proposes extract-and-evaluate CoT monitoring, where a quote extractor feeds a judge model, which raises catch rates of subtle side tasks by 18 to 27 points over prior monitors.
Finetunes models to hide backdoor-planning reasoning via ciphers or steganography, then shows the finetuned model monitors its own obfuscated chain-of-thought better than the base model, though gains partly match better prompts.
Shows that RL penalties on a model's final output suppress the same content in its chain of thought, so output-only monitoring can induce CoT obfuscation, with a mind-face separation partly mitigating it.
Argues that unfaithful chain-of-thought mostly reflects hidden hints gently biasing many small reasoning choices rather than deception, and that this limits how much misaligned scheming CoT monitoring would miss.
A multi-organisation position paper arguing that monitoring chains of thought is a valuable but fragile AI safety opportunity and should be preserved and studied as a safety agenda.
Decomposes LLM chain-of-thought into sentences and uses resampling, attention analysis and attention suppression to find pivotal 'thought anchors' that determine the final answer.
Interrupts chain-of-thought reasoning in LLaMA and GPT-4.1 to track how answer beliefs shift, finding noisy, non-monotonic trajectories and proposing belief tracking as a way to detect unfaithful reasoning.
Evaluates chain-of-thought monitors against sabotaging untrusted LLMs, finding CoT helps catch subtle sabotage but can be fooled by unfaithful reasoning, and proposes a hybrid CoT-plus-action monitor that improves detection.
Probes whether small Qwen models track and update beliefs about final answers during chain-of-thought, using sampling, prompting and linear probes; finds little evidence of anticipation, relevant to CoT faithfulness and monitoring.
Tests whether fine-tuning can reveal hidden secret computation in LLM chain-of-thought, finding no evidence of such circuits in small models and noting possible steganography directions.
Proposes an architecture that generates multiple English tokens per forward pass during chain-of-thought, to keep reasoning readable for oversight while reducing the information bottleneck, rather than switching to neuralese.
Proposes monitoring real-world LLM deployments, both internal and external, for covert scheming to estimate base rates, using classifiers and privacy-preserving analysis of stored trajectories.
Measures whether inference-time-compute reasoning models verbalise prompt cues that change their answers, finding much higher chain-of-thought faithfulness than non-ITC models, which matters for safety monitoring.