Research / Sources
AI research and discussion on LessWrong. www.lesswrong.com
Trains a learned black-box optimizer that automatically builds gridworld honeypots where PPO reward-hacks, outperforming CMA-ES and GP Bayesian optimization at finding cheating-eliciting levels.
Proposes RL training environments with deliberately misaligned helper agents so orchestrators must learn to detect and manage misalignment, making alignment part of the RLVR loop; discusses limits and second-order risks.
Argues that alignment in model weights does not transfer when agents delegate to weaker open models, so prompts and shared environment notes become a key risk channel for multi-agent swarms.
Argues that aligning AI to human moral judgments is misguided and proposes training agents to attune to societal flourishing and plurality, as AlphaZero learned chess without human games.
Proposes a paired gameable-versus-robust evaluation protocol for apparent-success-seeking, and a small experiment finds several frontier coding models contaminate few-shot prompts with test cases, some defying correction.
Argues that most misaligned AIs are partially misaligned, so negotiated deals could beat adversarial responses, and sketches the trust, verification and negotiation infrastructure needed to make such trades workable and safe.</reasoning_effort>
Speculative mechanistic-interpretability hypotheses on how grader-awareness features emerge from pretraining and RL, plus commentary on Anthropic's NLA and steering results on unverbalized grader awareness and reward-seeking.
Finetuning-based honesty elicitation on secret-side-constraint model organisms erases the hidden behaviour being audited, so apparent confessions are mostly hallucinated and the auditing method is fragile.
A philosopher argues that aligned values may not stay reflectively stable under moral skepticism at AGI/ASI levels, and proposes tying AI self-assessed welfare to morality as a way to keep alignment robust.
Compares RLHF, Constitutional AI and Deliberative Alignment through a persona-emotion-behavior framework, argues CAI yields more stable personas, and cautions against activation steering that models could learn to evade.
Argues that the Anthropic incoherence paper conflates three distinct agent failure modes (attention decay, insufficient information-seeking, and value miscalibration) and proposes tests to separate them.
Shows that a model can pass black-box alignment evaluations yet become arbitrarily misaligned after one small benign gradient step, both theoretically and via adversarially trained hair-trigger LLMs, arguing for white-box evaluation.
Argues that LLMs can now load human intent and values, so alignment is tractable but unsolved, citing InstructGPT, Constitutional AI, and emergent misalignment results.
Argues that training AI policies online creates risks, including alignment drift, hidden sabotage that evades detection, degraded activation monitors, and harmed capabilities, and proposes running online-trained and frozen models in parallel.
A comprehensive index of AI safety, alignment and control interventions, covering theory, formal methods, interpretability, oversight, robustness, control, and governance, with links to prior work.
Defines AI truth-seeking by its failure modes, including sycophancy and confirmation bias, proposes metrics for each, and tests bootstrap consistency, reasoning-coverage RL and Brier-reward RL as interventions.
Sketches a realistic target future where superintelligent AI stays under permanent human control through dynamic alignment targets, structured multi-source feedback, and institutional norms, without claiming a technical solution.
Proposes that models' misalignment may come from generalising sci-fi tropes of rogue AIs in training data, and suggests a 2x2 framing experiment to test whether dramatic narrative framing increases misaligned responses.