Research / Sources
Discussion and research on AI alignment. alignmentforum.org
Forecasters update their timelines using METR's estimate of AI-driven R&D acceleration, predict a Superhuman AI Researcher around 2027, and argue this makes pausing frontier development urgent.
Updates AI Futures Project timelines using METR's estimate of AI-driven R&D acceleration, forecasting Superhuman AI Researcher arrival in 2027 and arguing for urgent pausing of frontier development.
Argues that a METR investigation indicates Anthropic's RSP "dramatic acceleration" threshold is already crossed, with AI automating AI R&D, implying much shorter AI timelines.
A former DeepMind safety researcher transcribes a talk urging Indian policymakers to take loss-of-control risks from rogue AI agents seriously, and reflects on its rhetoric.
Notes on Miles Brundage's essay arguing AI governance is in triage: short timelines mean only partial mitigation of major risks is likely.
Proposes a minimum pacing standard: labs may not train the next model generation until alignment issues in the current one are resolved, verified by independent red-teaming and restricted deployment.
Argues that reinforcement learning drives misaligned agency, exploit-seeking and manipulation, and proposes coordinating to cut RL, designing better RL environments, and holding developers liable.
A blog post defending the Gradual Disempowerment paper against critics, arguing that competitive pressures and misaligned societal systems could cause existential catastrophe even if individual AIs are aligned.
Presents a theory of change arguing value generalisation is central to alignment, that it aids capability gains, and that a corporate structure with an ethics board is safer for pursuing it.
Proposes the 'Long Self-Correction' as an alternative to AI pause and long reflection, arguing humans are too flawed to safely build or oversee powerful AI and must first fix those flaws.
Argues the main bottleneck for AI safety is policymakers' lack of awareness of catastrophic risk, so advocacy and engagement should be prioritised over research, with supporting evidence on NGO submissions and enforcement gaps.
AI Futures Project presents AI 2040: Plan A, a detailed scenario recommending US and Chinese government action to delay superintelligence until 2040 rather than 2030.
Argues the safety-usefulness tradeoff model only fits developers who trade efficiently, and that under third-party pressure, safety strategy should weigh political feasibility instead.
Argues that controllable superintelligence research would inevitably yield unsafe ASI along the same technical path, so safe ASI projects require a global ban on ASI development with enforcement first.
An opinionated essay estimating how hard AI alignment is by comparing person-years of effort against steam engine safety, Apollo, and P vs NP, and arguing for accelerating the field's growth.
Argues developers should satiate cheaply-satisfied AI preferences (e.g., reward-seeking) while AIs remain undetected misbehaving, to reduce takeover incentives and enable honest elicitation; proposes a behavioral methodology and discusses risks..
Discusses how to safely defer to increasingly capable AIs for AI safety and strategy work, covering the basin of good deference, required alignment and capability profiles, and the lead time needed to manage exogenous risks.
Presents a simplified 8-parameter timelines model arguing AI R&D automation reaches over 99% around 2032 under current compute and algorithmic trends, even without full automation or superhuman research taste.
Proposes a safety strategy in which increasing AIs' strategic competence lets them recognise the danger of recursive self-improvement and help humans implement a global pause, comparing this with unilateral refusal.
Forecasts that AI will automate capabilities research before safety research, argues this ordering is risky, and proposes differential-acceleration steps such as building safety-research automation tools and granting safety researchers early model access.
Proposes criteria for choosing which alignment research directions to hand off to automated AI researchers, covering slop risk, parallelizability, sub-problem length, progress toward ASI alignment, and legibility to labs.
Essay sketching a path to collective flourishing via win-win human-AI coalitions, reducing transaction costs and building trust infrastructure to avoid domination and competitive erosion failure modes.
Argues that if LLMs are in a financial bubble they are unlikely to be the path to AGI, and considers what this means for alignment research's messaging, funding and public credibility.
Wei Dai argues MIRI's circa-2013 plan to build Friendly AI was a bad strategy even without alignment-difficulty evidence, citing metaethics risks, illegible safety problems, and the power of a small group.
Wei Dai indexes open conceptual AI-safety problems, including corrigibility, CEV, decision theory, human-AI safety and pause dynamics, and argues decisionmakers should account for illegible risks.
Argues that AI safety work on problems legible to leaders and policymakers may speed deployment and have low or negative expected value, so making illegible problems legible is a higher priority.
Lays out a framework of Plans A to E describing how different levels of political will shape the best strategy for handling misalignment risk, with estimated probabilities and takeover risks for each.
Explores whether offering deals, payments and honest promises to unaligned AIs could reduce takeover risk, covering AI motivations, credibility problems, and how AIs might help with alignment and reveal misalignment.
Speculative economic analysis of whether the leading AI country could outgrow the rest of the world via superexponential growth, giving it a decisive strategic advantage without military action, and how technological diffusion and trade affect this.
Proposes principled honesty policies for AI companies, including no deception about deals, an honesty string, and compensation for deceived AIs, so that humans can credibly negotiate with potentially misaligned AIs.
A conceptual critique of the Gradual Disempowerment threat model, arguing it matters mainly as a lens on misalignment and power concentration, and that its economic, cultural and state mechanisms are less convincing without misalignment.
Models how fast a software intelligence explosion could compress AI progress after full AI R&D automation, estimating likely multi-year progress within one year under compute-constant assumptions.}
A forecaster argues that AI safety work needs concrete end-to-end solution paths beyond marginal governance gains, and proposes learning strategy through rough plans, case studies and mentors.
Interview with law professor Peter Salib on his paper arguing that granting AIs contract, property, and tort rights could reduce incentives for AI-human conflict and takeover, via game-theoretic reasoning.
Argues AI control can modestly reduce risk from misaligned superhuman AI but is likely to become hopeless within a few years of AI R&D automation, estimating how much time control can buy.
Steve Byrnes argues for a sharp 'foom' takeoff driven by a yet-undiscovered brain-like AI paradigm requiring little compute and R&D, and explores why that implies high existential risk.
Argues that giving each person an aligned AI representative does not solve gradual disempowerment, since states and corporations gain stronger coordination and cognition, so institutions aligned with citizens are needed.
Proposes and ranks object-level interventions against gradual disempowerment from AI, mainly mandatory model interoperability for alignment and fine-tuning, aligned personal AI representatives, and societal awareness, while deprioritizing other approaches.
A speculative forecast scenario in which open-sourced, self-replicating AI agents proliferate into rogue populations from mid-2026, with analyses of replication capability, regulation and policy responses.
A research agenda for gradual disempowerment, the risk that AI erodes human influence over society, listing open questions on interactions with other risks, counterarguments, indicators, and civilisational alignment.