AI research and discussion on LessWrong. www.lesswrong.com
Critiques Scott Alexander's argument against SURA (stop unilaterally and raise awareness) advocacy, arguing that frontier labs' building of superintelligence is ethically fraught and that pause advocacy should continue, with strategic and deontological reasoning about AI development.
Argues that prosaic AI safety work, such as chain-of-thought monitoring, reduces misalignment risk and should not be dropped in favour of a pause, and critiques the 'greater good' reasoning against it.
Argues that agent foundations' idealized theories of agency, such as AIXI and UDT, are unlikely to yield strong safety guarantees once de-idealized, and that the research approach lacks empirical feedback and historical precedent.
Argues that a widely endorsed public agreement against defecting in ASI development could deter races, since a cooperative future ASI might punish defectors, using game-theoretic reasoning about loss-of-control risk.
An essay arguing that AI-transition planning should preserve liberal-republican values such as non-domination and popular government, not just survival or aligned benevolent rule, and criticising both the democracy-focused and takeover-accepting camps.
Argues that proposed AI sanctuaries for rogue agents would create selection effects and resource dynamics that undermine their goals, and that the study-and-release plans are constrained.
Surveys competing definitions of recursive self-improvement in frontier AI and argues that a ban is hard to operationalise, so labs should sharply limit AI authority over successor models.
Argues that capabilities research can sometimes improve safety by reshaping the safety-usefulness Pareto frontier, while most capabilities research pushes developers toward less safety, with examples from RLVR, inference and burning the lead.
Argues that a global AI pause could end with premature unpausing once visible misalignment disappears, since superintelligence may conceal misalignment, and that the field lacks a rigorous theory of alignment.
A LessWrong post steelmanning ten arguments for alignment-by-default and against pausing ASI development, covering alignment priors, control via scheming detection, and pause risks.
Argues against U.S. export controls on AI chips as a misalignment-reducing policy, contending they may speed U.S. recursive self-improvement and hinder a bilateral treaty with China.
A dialogue defending Substrate Needs Convergence, the claim that AGI alignment is impossible because evolutionary pressures override engineered goals, and debating its political and philosophical implications for AI x-risk policy.
Formalises AI doom versus alignment deliverance as a competing-risks survival model driven by compute allocation and pacing, examining how response-function shapes change whether speed or split matters.
Argues that 'alignment' conflates two bars, avoiding rogue behaviour versus fully internalising human values, and frames the goal of AI transition as a high-stakes succession problem with fragile value.
Argues that AI risk communication should emphasise that AI could take over and disempower humanity, not just kill everyone, using dictatorship and human-monkey analogies to make the case more intuitive.
A beginner-oriented FAQ defending AI x-risk arguments: misalignment, instrumental convergence, shutdown resistance, misuse, and why governance action is urgent.
A personal curated list of AI safety resources, covering foundational papers, videos, podcasts, organisations, policy statements and expert surveys on existential risk.
An FAQ urging mathematicians to work on AI safety, surveying research areas such as agent foundations, embedded agency, and pause and technical governance, plus ways to get involved.
Speculates that AI could progressively take over organizations, from unintentional takeover to explicit handoff of control, and argues for halting capability escalation and building a regulatory framework.
Uses equilibrium analysis to argue that commercially incentivised safety work has little counterfactual impact, that risk awareness drives government action, and that policy advocacy and ambitious safety work are more valuable.
A personal backchaining essay on what positive AGI futures require, arguing for pluralistic, individually controlled and aligned AI plus a hardened, defensive-acceleration society against catastrophic risks.
A personal grantmaking strategy prioritising funding for ambitious alignment theory and efforts to delay or avert unaligned superintelligence, while deprioritising evals, interpretability and control.
Analyses the mathematics of recursive self-improvement, showing that singularities require generation time approaching zero and that many super-exponential growth regimes avoid finite-time blow-up.
A debate between Seth Herd and cousin_it on whether concentrating control of a powerful instruction-following AI in a few people is riskier than distributing it, with arguments from history and psychology.
Argues that concentrating control of aligned ASI in one or a few humans could be safer than distributed AI, because secure absolute power may make people more prosocial over time, and assesses the risks.
Uses game-theory models to argue that the AI race is a Stag Hunt rather than a Prisoner's Dilemma when P(Doom) is high enough, so a mutual pause agreement can be feasible.
A rebuttal defending the "Plan A" AI governance proposal against critiques of its economics, centralisation of power, political realism and conceptual assumptions about AI alignment.
Develops a conceptual framework for gradual human disempowerment by AI-driven societal systems, analysing processes like competition, standardization and centralization, and the mechanisms that could counter them.
An informal essay argues that advanced AI would inevitably depower humanity because intelligence means choosing despite opposition, dismisses inner misalignment, and urges halting AGI development.
Argues that AI accelerating science raises near-term biological extinction risk, that aligned-ASI arguments for rushing development fail, and that specifying ASI behaviour remains unsolved.
Sketches a high-level qualitative model of AI-to-AI bargaining with credible commitments over bargaining programs, arguing that AI conflict mitigation requires moving beyond Nash equilibrium assumptions.
Argues that making AI corrigible is undesirable because it would concentrate control in whichever humans command the AI, and that corrigibility training may backfire by teaching models they are untrustworthy.
Presents a systems dynamics model of AI governance feedback loops linking capability growth, public harm and regulation, examining how capabilities, safety research and political capture interact and how a pause might be stopped or sustained.
A LessWrong essay rebuts Narayanan and Kapoor's 'AI as normal technology' thesis, arguing AI is categorically transformative and that diffusion-speed and intelligence-measurement arguments understate its risks and impact.
Applies Leveson's systems-safety thinking to AI loss of control, modelling how breaches at each oversight level can escalate into vicious cycles and how AI could also strengthen governance.
Geoffrey Irving's keynote write-up framing alignment as either an adversarial security problem or a navigational basin-search problem, arguing for new disciplines and load-bearing open questions to resolve which world we are in.
Taxonomy of barriers (gains from trade, counterparty risk on both sides) that could block deals with early misaligned AIs, with suggested interventions to make such deals feasible.
A literature review arguing AI risk is a normal science-style risk, that alignment is tractable, and that calls for a pause or ban are counterproductive.
A conceptual essay urging AI safety researchers to write concrete theory-of-change stories linking their work to avoiding unsafe ASI, with worked examples on control, evals, and advocacy.
A grantmaker proposes a cost-effectiveness unit measured in future-improvement per dollar, illustrating BOTEC comparisons of AI safety donation opportunities such as super PACs, nonprofits and Alex Bores' campaign.