Takes on Automating Alignment
Argues that alignment research should be converted into hill-climbable tasks that AI models can automate, and discusses reward hacking risks and mitigations such as held-out validation sets and honeypots.
Research / Sources
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Alignment
Date Any time
Product Alignment is not Superintelligence Alignment (and we need the latter to survive)
Argues that making products like Claude friendly is not the same as solving superintelligence alignment, and warns that conflating the two gives a false sense of security about safe future AI.
Brain-inspired LLM alignment
Proposes a brain-inspired LLM architecture with multiple assessor heads that score thoughts, trained with AI-labelled RL from early in pretraining, to improve alignment and resist reward hacking.
What Success Might Look Like
Sketches a realistic target future where superintelligent AI stays under permanent human control through dynamic alignment targets, structured multi-source feedback, and institutional norms, without claiming a technical solution.
Recursive alignment with the principle of alignment
Proposes recursive alignment, defined as AI aligning with the process of resolving goal conflicts, and argues it could resist power-seeking and deceptive alignment, critiquing deliberative alignment.
Training AI to do alignment research we don’t already know how to do
Argues that AI can safely automate alignment research if agents are trained for truth-seeking, with humans nudging rather than grading ground truth, provided agents stay non-egregiously misaligned.
Constitutional Classifiers: Defending against universal jailbreaks (Anthropic Blog)
Introduces Constitutional Classifiers, input and output classifiers trained on synthetic constitution-based data that filter the vast majority of universal jailbreaks with minimal over-refusal and moderate compute overhead.