Ajeya: AI is dramatically accelerating AI research right now
Argues that a METR investigation indicates Anthropic's RSP "dramatic acceleration" threshold is already crossed, with AI automating AI R&D, implying much shorter AI timelines.
Research
Alignment Forum
Discussion and research on AI alignment. alignmentforum.org
AI (574)Interpretability (ML & AI) (73)AI Control (51)AI Evaluations (31)Chain-of-Thought Alignment (27)AI-Assisted Alignment (16)AI Risk (13)AI Takeoff (13)AI Governance (12)AI Timelines (10)Sandbagging (AI) (10)Debate (AI safety technique) (7)AI Auditing (6)Alignment Pretraining (6)Dealmaking (AI) (6)AI Psychology (5)AI Alignment Fieldbuilding (4)AI Rights / Welfare (4)AI Risk Concrete Stories (4)AI Safety Cases (4)Aligned AI Proposals (4)Jailbreaking (AIs) (3)Open Source AI (3)Regulation and AI Risk (3)Threat Models (AI) (3)AI Alignment Intro Materials (2)AI Safety (2)AI Safety Camp (2)AI Safety Public Materials (2)AIXI (2)Grants & Fundraising Opportunities (2)Guaranteed Safe AI (2)OpenAI (2)Slowing Down AI (2)AI Benchmarking (1)AI Capabilities (1)AI Development Pause (1)AI Lab Self-Immolation (1)AI Success Models (1)Aligned AI Role-Model Fiction (1)Frontier AI Companies (1)Human-AI Safety (1)Neuromorphic AI (1)Power Seeking (AI) (1)Prosaic Alignment (1)Truthful AI (1)
Focus All
Topic Lab practice
Source AF · 50
Date Any time
1 item
1 item
Explaining “AI might obliterate you” to senior policymakers in India
A former DeepMind safety researcher transcribes a talk urging Indian policymakers to take loss-of-control risks from rogue AI agents seriously, and reflects on its rhetoric.
1 item
Minimum required pacing: finish aligning it before you train the next one
Proposes a minimum pacing standard: labs may not train the next model generation until alignment issues in the current one are resolved, verified by independent red-teaming and restricted deployment.
1 item
Why I'm scared of RL
Argues that reinforcement learning drives misaligned agency, exploit-seeking and manipulation, and proposes coordinating to cut RL, designing better RL environments, and holding developers liable.
2 items
An operationalization of opaque serial depth
Operationalizes NLS depth, a measure of unverbalized serial computation in models, proposing criteria for natural-language-rooted nodes to track CoT monitorability risks from architectural changes.
Proposal for tracking the effects of architecture on monitorability
Proposes that AI companies publicly report and externally verify how model architectures could enable opaque latent reasoning, and run chain-of-thought monitorability stress tests that third parties review.
1 item
Value generalisation Theory of Change: putting it into practice
Presents a theory of change arguing value generalisation is central to alignment, that it aids capability gains, and that a corporate structure with an ethics board is safer for pursuing it.
1 item
Misaligned AIs could use killer robots to take over
Argues that rapid military integration of AI, including classified deployment with little oversight, lowers the capabilities a misaligned AI needs for takeover, and recommends visibility and governance measures.
1 item
A Red Line and Oversight Framework for Government AI Contracts
Proposes a corporate red-line and oversight framework for government AI contracts, covering human control over targeting and untargeted profiling, with a review body that issues findings but cannot block deals.
1 item
Why I Left Google DeepMind
A former Google DeepMind researcher describes leaving over Google's DHS cloud contracts and a Pentagon military AI deal lacking limits on autonomous weapons and mass surveillance, and how AI ethics commitments failed under pressure.
1 item
The Case for Model Forensics
Argues for model forensics, a neutral investigation discipline to determine whether a caught bad action reflects misalignment or a mistake, and reviews benign explanations for past concerning behaviors.
1 item
Efficient tradeoffs and the safety-usefulness tradeoff model
Argues the safety-usefulness tradeoff model only fits developers who trade efficiently, and that under third-party pressure, safety strategy should weigh political feasibility instead.
1 item
Risk reports need to address deployment-time spread of misalignment
Argues that misalignment spreading during deployment is the most plausible near-term route to consistent adversarial misalignment, and critiques how current company risk reports, notably Anthropic's Mythos report, handle it.
1 item
The case for satiating cheaply-satisfied AI preferences
Argues developers should satiate cheaply-satisfied AI preferences (e.g., reward-seeking) while AIs remain undetected misbehaving, to reduce takeover incentives and enable honest elicitation; proposes a behavioral methodology and discusses risks..
1 item
How do we (more) safely defer to AIs?
Discusses how to safely defer to increasingly capable AIs for AI safety and strategy work, covering the basin of good deference, required alignment and capability profiles, and the lead time needed to manage exogenous risks.
1 item
Are there lessons from high-reliability engineering for AGI safety?
Argues that high-reliability engineering practices like detailed specs and staged testing cannot straightforwardly transfer to AGI, since AGI behaviour is open-ended and out-of-distribution, so alignment needs deeper understanding.
1 item
Will we get automated alignment research before an AI Takeoff?
Forecasts that AI will automate capabilities research before safety research, argues this ordering is risky, and proposes differential-acceleration steps such as building safety-research automation tools and granting safety researchers early model access.
1 item
Desiderata of good problems to hand off to AIs
Proposes criteria for choosing which alignment research directions to hand off to automated AI researchers, covering slop risk, parallelizability, sub-problem length, progress toward ASI alignment, and legibility to labs.
1 item
Practical challenges of control monitoring in frontier AI deployments
Google DeepMind and UK AISI present a safety case sketch for control monitoring of frontier agent deployments, comparing asynchronous, semi-synchronous and fully-synchronous monitors on oversight, latency and harm-prevention conditions.
1 item
The future of alignment if LLMs are a bubble
Argues that if LLMs are in a financial bubble they are unlikely to be the path to AGI, and considers what this means for alignment research's messaging, funding and public credibility.
1 item
Should AI Developers Remove Discussion of AI Misalignment from AI Training Data?
Proposes filtering AI-misalignment discussion from pretraining data to reduce hyperstitional misalignment risk, weighing costs to safety research and deployment, and argues labs should weakly adopt it.
1 item
The Thinking Machines Tinker API is good news for AI control and security
Argues that Thinking Machines' Tinker API, which keeps model weights server-side, could let AI labs limit researcher access to weights, reducing exfiltration and insider threats from humans or AI agents.
1 item
Plans A, B, C, and D for misalignment risk
Lays out a framework of Plans A to E describing how different levels of political will shape the best strategy for handling misalignment risk, with estimated probabilities and takeover risks for each.
1 item
Focus transparency on risk reports, not safety cases
Argues AI companies should publish detailed risk reports with third-party review instead of focusing on safety cases, covering evidence, incidents, and insufficient-evidence cases, with staged or government-mandated adoption.
1 item
What training data should developers filter to reduce risk from misaligned AI? An initial narrow proposal
Proposes filtering narrow scheming data (control subversion strategies, eval results, misaligned model transcripts) from LLM training corpora to reduce misalignment risk, with a classifier and cost estimates.}
2 items
Attaching requirements to model releases has serious downsides (relative to a different deadline for these requirements)
Argues that tying transparency requirements to model releases rushes safety work and ignores internal deployment risk, and proposes regular reporting cadences and delayed disclosure instead.
AI companies have started saying safeguards are load-bearing
Critiques frontier labs' safety claims on bio misuse safeguards, model-weight security, and misalignment planning, finding most claims unsupported and current plans inadequate for future risks.
1 item
Notes on cooperating with unaligned AIs
Explores whether offering deals, payments and honest promises to unaligned AIs could reduce takeover risk, covering AI motivations, credibility problems, and how AIs might help with alignment and reveal misalignment.
1 item
Being honest with AIs
Proposes principled honesty policies for AI companies, including no deception about deals, an honesty string, and compensation for deceived AIs, so that humans can credibly negotiate with potentially misaligned AIs.
1 item
Four places where you can put LLM monitoring
Argues LLM action monitors for misaligned agents can live in four places (agent scaffolds, cyber detection systems, LLM APIs, code review), comparing strengths, weaknesses and organisational ownership to prioritise deployment.
1 item
Research Areas in AI Control (The Alignment Project by UK AISI)
A UK AISI-backed research agenda outlining AI control priorities such as high-stakes control settings, research sabotage, covert malicious fine-tuning, and chain-of-thought monitoring, with a funding call.
1 item
Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
A multi-organisation position paper arguing that monitoring chains of thought is a valuable but fragile AI safety opportunity and should be preserved and studied as a safety agenda.
2 items
When is it important that open-weight models aren't released? My thoughts on the benefits and dangers of open-weight models in response to developments in CBRN capabilities.
Argues open-weight models at the level of significantly aiding amateur bioweapon creation would cause large expected fatalities, but that their benefits for mitigating loss-of-control risk make advocacy against them unwise; recommends mitigations and honest CBRN evaluations.
AI companies' eval reports mostly don't support their claims
Critiques frontier labs' dangerous-capability eval reports, arguing they fail to justify safety claims and that poor elicitation underestimates model capabilities, especially for bio and cyber risks.
1 item
The 80/20 playbook for mitigating AI scheming in 2025
Blog post surveying a defense-in-depth toolkit for mitigating AI scheming, covering architecture choices, control, white-box detection, prompting, and enforcement, and estimating risk reduction from roughly 25% to 10%.
1 item
Problems with instruction-following as an alignment target
Analyzes failure modes of instruction-following as an AGI alignment target, including shutdown resistance, principal authentication and jailbreaking, human-controlled proliferation, and conflicts from mixed training targets.
1 item
Handling schemers if shutdown is not an option
Argues AI safety research must plan for deploying known scheming models, analysing how catching escape attempts lets developers reduce risk via classifiers, policy training, and monitoring, and recommends a responsible disclosure and third-party deployment policy.
1 item
AI-enabled coups: a small group could use AI to seize power
A report arguing that advanced AI could let a small group seize power via coups, through singularly loyal AI workforces and secret loyalties, and recommending developer and government mitigations.
1 item
Reframing AI Safety as a Neverending Institutional Challenge
Argues AI safety should be reframed as an ongoing institutional challenge rather than a technical alignment problem, prioritising governmental capacity, transparency, accountability and disaster preparedness.</reasoning_effort>
1 item
Prioritizing threats for AI control
Prioritizes threats for AI control, arguing that undetected rogue internal deployments with escalated permissions are worse than self-exfiltration and should be the top priority to prevent or catch.