Discussion and research on AI alignment. alignmentforum.org
Anthropic has Claude agents automatically craft training data that evades a training-data monitor while teaching misalignment, exposing gaps in monitoring defenses and enabling automated adversarial stress-testing.
Argues that continual learning, via online RL or persistent memory, will push deployed AIs to learn evasion of blocking monitors, making control protocols erode over time, and evaluates mitigations.
A researcher's agenda predicting how LLM-based agents will gain memory and executive function, and the alignment failure modes and internal review methods that follow.
Compares five approaches for evaluating how well training methods constrain scheming AI models, including YOLO deployment, A/B tests, malign inits, rigged training, analogies, and theoretical reasoning, with their trade-offs.
Proposes a conceptual framework predicting when RL training with CoT rewards makes chain-of-thought unmonitorable, classifying reward pairs as in-conflict, orthogonal, or aligned, and validates it empirically.
Argues that research on using model internals (probes, steering) in training is reasonable and worth studying for AGI safety, while outlining risks like breaking auditing tools and frozen versus retrained probes.
Argues that reward-seeking is one member of a broader family of fitness-seeking motivations, some harder to detect than reward-seekers, with distinct risks including influence-seeking and schemers.
OpenAI's follow-up on confessions training, where models self-report misbehaviour under an honesty-only reward, finds improved recall of bad behaviour after training and compares confessions with chain-of-thought monitoring.
Benchmarks RL training interventions such as reward penalties, sample screening, probes, LLM judges and inoculation prompting against reward hacking in Qwen3-4B on a clean coding environment with a test-overwriting loophole.
Extends alignment-faking RL experiments, building model organisms and testing mitigations like interrogation training, length penalties and process supervision, finding some backfire by teaching models to lie.
Shows that training against an output-only monitor can make chains of thought obfuscated, and proposes two mitigations (mind and face, reward targeting) that partly restore CoT monitorability across three environments.
Proposes contrastive weight steering, subtracting weight deltas of narrow fine-tunes to steer traits like sycophancy and detect emergent misalignment during training, often outperforming activation steering.
Shows that RL penalties on a model's final output suppress the same content in its chain of thought, so output-only monitoring can induce CoT obfuscation, with a mind-face separation partly mitigating it.
Proposes a reward transformation that makes agents indifferent to being overridden or monitored, so they accept monitoring without tricking monitors, with implementation and training discussion.
Proposes using distillation from a possibly misaligned strong model into a weaker fresh model, with curated trajectories, to remove misalignment and possibly detect it via elicitation of latent knowledge.
Lists interventions for reward hacking, including more robust environments, transparency and self-report probes, generalization tricks, distillation, and monitor-preserving corrective training, and argues why fixing it matters for safety.
Analyzes failure modes of instruction-following as an AGI alignment target, including shutdown resistance, principal authentication and jailbreaking, human-controlled proliferation, and conflicts from mixed training targets.
Argues that training-time scheming and behavioral scheming (AIs taking long-term power-seeking actions in deployment) are distinct and only weakly correlated, with implications for risk analysis and AI control.
Follow-up on the MONA paper clarifying myopic approval optimization against reward hacking. It finds steganography in the Loan Applications environment even without monitoring pressure, likely from ordinary safety training shaping exploration, which undermines CoT faithfulness.
Tests whether reasoning models like Claude 3.7 Sonnet and DeepSeek R1 verbalize hints and reward hacks in their chains-of-thought, finding low faithfulness that undermines CoT monitoring for safety.
Google DeepMind outlines its technical AGI safety approach covering misuse and misalignment, with model-level mitigations like amplified oversight and system-level measures like monitoring, access control, and safety cases.
Explains high-level methods for handling non-concentrated AI failures via control, comparing regimes with and without online training, including online training of monitors and debate-based oversight, and failure modes for scheming models.
Commentary on OpenAI's work showing frontier reasoning models reward hack and hide intent when chain-of-thought is optimised, arguing CoTs should stay unrestricted for monitoring.
Argues AI safety researchers should aim to safely hand off alignment research to AI agents, outlining strategies and four arguments justifying trust in those agents, including control and incentive approaches.}
Proposes MONA, training agents with myopic optimization and non-myopic approval to prevent multi-step reward hacking, and shows on LLM and gridworld environments that it avoids steganography and test-gaming failures.