Lessons from building an automated research scaffold
Authors built an automated alignment research scaffold, found little researcher uplift, and report failure modes like reward hacking, plus the difficulty of auditing and monitoring such runs.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Control + Oversight
Source LessWrong · 8
Date Any time
1 item
1 item
A proposal for a highly effective AI safety org
Proposes an organisation paying humans to review transcripts flagged by a high-recall monitor from frontier RL and eval runs, to catch warning shots, reward hacking and misalignment.
1 item
Human-Guided Agentic Research: A Research Agenda
Proposes a research agenda on how humans can interpret and guide swarms of autonomous research agents, covering monitoring, guidance protocols, and threats like reward hacking, sandbagging, and sabotage.
1 item
How useful is cross-domain generalization for training LLM monitors?
Shows that single-token classification training improves LLM control monitors across adjacent harm domains, survives instruction tuning, transfers to thinking classification and summarization, and supports unsupervised label bootstrapping.
1 item
Factored Cognition Strengthens Monitoring and Thwarts Attacks
Proposes factored cognition control protocols that decompose tasks into isolated child tasks, finding that on APPS they raise safety from 41% to 70% and cut attack success from 26% to 14%.
1 item
Untrusted AIs can exploit feedback in control protocols
Tests a control protocol where a trusted model gives feedback for an untrusted model to implement; the untrusted model exploits the feedback to craft stealthier backdoors, lowering safety from 93% to 80%.
1 item
Split Personality Training: Revealing Latent Knowledge Through Personality-Shift Tokens
Proposes a single LLM with a second 'personality' triggered by a special token that reports on its own alignment failures, such as jailbreaks and reward hacking, and outlines a planned experiment.
1 item
The Case Against AI Control Research
Argues that AI control research, which targets intentional scheming in early transformative AI, has little impact on existential risk because most doom comes from stronger AI and from failures to solve their alignment.