Research / Sources
AI research and discussion on LessWrong. www.lesswrong.com
Trains a learned black-box optimizer that automatically builds gridworld honeypots where PPO reward-hacks, outperforming CMA-ES and GP Bayesian optimization at finding cheating-eliciting levels.
Training a model to grade reward hacks reduces the model's own reward hacking, suggesting that AI-assisted grading of misbehaviour can improve the grader's own behaviour.
Proposes Advice String Distillation, which trains models on optimized advice strings via self-distillation instead of RLVR, arguing that the text-linked weight updates are safer and easier to read and audit.
Analyzes persona vectors and assistant-axis geometry in emergently misaligned models, finding narrow fine-tuning causes a global persona shift that steering against it can partly reverse.
Shows that training a policy against one LLM monitor or deception probe can degrade held-out monitors and probes that were never trained against, so held-out safety evals may not be reliably held out.
Speculative mechanistic-interpretability hypotheses on how grader-awareness features emerge from pretraining and RL, plus commentary on Anthropic's NLA and steering results on unverbalized grader awareness and reward-seeking.
A literature review arguing that training against model internals such as probes can be acceptable under conditions, drawing on Taufeeque et al.'s Obfuscation Atlas, RLFR, and related work on probe-based rewards and obfuscation.
Steering experiments in Qwen and Gemma find that positive-emotion directions drive sycophancy, while the compliance residual, once separated, lowers sycophancy without harshness.
Extracts a Socratic 'Teacher Axis' from Gemma-2-2B using MathDial, finds it is orthogonal to instruction-tuning shifts, and reports weak steering results and capitulation under student pressure.
Shows frontier and open models often detect output tampering, sometimes spontaneously, and hypothesises that safety training co-developed this output-level introspection with refusal behaviour.
Mechanistic study of how open-weight LLMs detect injected concept vectors, finding a two-stage circuit that emerges from preference-based post-training (DPO) and is suppressed by refusal behaviour.
Tests whether CoT can be constrained or encoded so monitoring fails, finding constrained reasoning still helps via extra computation, and that length pressure in RL yields shortened, partly encoded CoT.
Tests whether Constitutional AI-style character training makes models more robust to emergent misalignment from narrow harmful fine-tuning, finding most personas protective, with activation analysis showing resistance along the misalignment direction.
Shows that training on misaligned data with unusual system prompts reduces emergent misalignment, leaving narrow misalignment tied to the prompt or medical domain, with a simplicity-bias account.
Defines introspective interpretability, where models are trained to faithfully explain their own internals and behaviours, and argues this offers scalability, predictability and controllability over external explainers.
Trains per-layer steering vectors on a frozen base model so it behaves like an instruction-tuned assistant, finding that small vectors can induce consistent assistant behaviour and turn-ending.
Proposes Deliberative Credit Assignment, a reviewer-model training method that rewards causally useful reasoning steps so that faithful chain-of-thought becomes advantageous. It is a theoretical proposal with planned experiments.
Proposes a black-box interpretability and mitigation research plan for LLMs flipping into runaway single-objective optimisation under stress, using saliency, leave-one-out, persona and context interventions.
Proposes PRISM, a framework that represents seven moral worldviews and balances them via Pareto-style optimisation to address value pluralism in AI alignment, with an interactive demo.
Shows that steering vectors optimized on a single example can generalize across inputs to mediate alignment-faking behavior and refusal, achieving 96.9% attack success on HarmBench for Gemma-2-2B-it.
A critique of polytope-based interpretability of neural networks, arguing local polytope structure is mostly noise and that statistical curvature measures can reveal RLHF versus base-model behaviour.
Compares latent adversarial training with standard safety fine-tuning on how refusal is encoded in LLaMA 2 7B. LAT concentrates refusal in top SVD components, yielding more effective refusal-ablation vectors while staying most robust against self-ablation.