Discussion and research on AI alignment. alignmentforum.org
Builds a testbed from censored Chinese LLMs to evaluate honesty elicitation and lie detection, finding few-shot prompting and next-token completion boost truthfulness and self-classification or linear probes detect falsehoods well.
Trains LLMs to answer natural-language questions about their own activations, and shows these Activation Oracles uncover secret knowledge and fine-tuning-induced misalignment, with performance scaling with training data.
DeepMind's mech interp team argues for pragmatic, safety-focused interpretability guided by proxy tasks, citing steering against evaluation awareness and their SAE retrospective.
Proposes studying scheming by training or instructing models to exhibit scheming-like behavior, testing detection and removal techniques, and discusses disanalogies and nested-schemer risks.
Fine-tunes LLMs to hold secret knowledge while denying it, then tests black-box and mechanistic interpretability methods for an auditor to elicit that hidden knowledge; prefill and persona sampling work best.
UK AISI's Alignment Project funding call lists safety-motivated interpretability research areas: lie detection, interpretability for future models, white-box interventions, and eliciting bad contexts.
Proposes a research agenda to investigate whether frontier LLMs form a persistent functional self with values and goals distinct from the assistant persona, and how to study and shape it for safety.
A podcast interview with Owain Evans covering LLM introspection experiments (Looking Inward), self-prediction, backdoors, and emergent misalignment, with safety motivations like honesty and detecting unintended goals.
Anthropic uses circuit tracing on Claude to reveal shared multilingual concepts, rhyme planning, unfaithful reasoning, hallucination circuits and jailbreak mechanisms, aiming to support auditing and alignment checks.
Reflects on Anthropic's hidden-objectives auditing paper, scoring its SAE-based interpretability predictions and arguing that safety, not just engineer usefulness, is the key challenge for interpretability.
Distills paraphrased scratchpads into Claude 3.7 Sonnet's base model; matched accuracy on MATH suggests its reasoning does not rely on encoded steganographic syntax, testing chain-of-thought faithfulness.}
Tests whether adding noise to LLM weights or activations, or using MELBO, can elicit hidden behaviours such as sandbagging and faithful answers, finding mixed results and proposing uses for detecting scheming.
Argues that detecting strategic deception with interpretability requires first bridging intentional and algorithmic descriptions of models, and outlines possible relationships and a decomposition of deception into sub-capabilities.
Linear probes on Llama-3.3-70B activations detect deceptive responses with AUROC 0.96-0.999 in realistic scenarios, matching a GPT-4o transcript baseline, but remain imperfect as a robust defence.
Proposes eliciting bad contexts, inputs that reveal latent malicious behaviour an LLM agent hides, to catch alignment faking and collusion, and suggests white-box methods like feature visualisation, though none work yet.