Research / Sources
AI research and discussion on LessWrong. www.lesswrong.com
Trains a learned black-box optimizer that automatically builds gridworld honeypots where PPO reward-hacks, outperforming CMA-ES and GP Bayesian optimization at finding cheating-eliciting levels.
Tests whether Natural Language Autoencoder readouts detect steganographic secrets; finds they miss internalized (SFT) behaviors that remain causally active, suggesting NLA monitors may fail to detect trained-in deception.
Shows that DeepSeek-V3 J-lenses targeting the final layer inherit a dominant output-specific direction causing language shifts, kurtosis artifacts and eval inflation, and recommends the penultimate layer as a default target.
Black-box experiments with input caching and filler tokens suggest an LLM performs belief-propagation-like probabilistic inference internally without chain-of-thought on logic and error-correction tasks.
Proposes a research agenda and list of 109 open problems in mechanistic interpretability of biological foundation models, motivated by biosecurity auditing of dangerous learned biology.
Shows that interpretability results on model organisms depend strongly on training method, so single model organisms are unreliable benchmarks for auditing safety properties in LLMs.
Empirical null result: behavioral tests on Hughes et al.'s open-source Llama 70B checkpoints found no strict alignment faking, so the planned probe generalization test could not be run.
Trains a toy ReLU network under L4 loss that performs computation in superposition, then reverse-engineers its binary-codeword mechanism and validates it with a three-scalar description, providing a ground-truth testbed for parameter decomposition methods.
Studies clustering and collapse dynamics of token representations in transformers (GPT-2, ALBERT, BERT) with trained versus random weights, finding trained models resist the collapse predicted by the attention-as-particle theory.
Argues that apparent introspective reports under concept injection in open models are better explained by entropy shifts and concept leakage than by genuine self-knowledge, based on logit, mismatch and inversion tests.
UK AISI gives Qwen3 models tools to self-steer their internal states with steering vectors, finding preferences, partial introspection, and self-medication under frustration, relevant to self-modification risks.
UK AISI report arguing that current AI oversight via auditing, CoT monitoring and incident investigation rests on fragile properties likely to degrade, and recommending mitigations and research directions.
Probes and activation patching show a sudoku-solving transformer linearly encodes board state by substructure (rows, columns, boxes) rather than by cell, and uses it for prediction. This is non-safety interpretability of a puzzle-solving model.
Replicates Anthropic's steering-vector suppression of evaluation awareness on GLM-5 blackmail scenario and finds semantically unrelated control vectors produce effects as large as designed ones, undermining their use as baselines.
Applies Thought Anchors resampling, causal masking, receiver heads and early-stopping probes to LLM legal reasoning on ambiguous criminal cases, finding judgment shifts late in reasoning. Mainly mechanistic analysis without a clear safety framing.
Evaluates whether GPT-OSS-20B grid-world agents are goal-directed by combining behavioural tests with probes of internal cognitive maps and plans, finding imperfect but real goal-directedness and goal-like distractor vulnerabilities.
Shows cross-layer transcoders can learn unfaithful shortcut circuits that skip intermediate computation in a toy model, argues sparsity penalties incentivize this, and gives preliminary evidence in GPT-2 and Gemma 2.
Probes steering-vector activations across 15 open-weight models from 0.27B to 70B to show evaluation awareness grows with model size, with a power-law trend and Qwen as an outlier.
Formal analysis of minimal, agent-agnostic world models for sandboxing AI agents in safety evaluations, showing an efficiency-versus-interpretability trade-off and proposing belief-based alternatives.
Shows that steering vectors optimized on a single example can generalize across inputs to mediate alignment-faking behavior and refusal, achieving 96.9% attack success on HarmBench for Gemma-2-2B-it.