1000 Years of AI Safety
Models how much AI safety research a multi-year pause among top labs could produce, including verification bottlenecks and Plan A scaling tradeoffs, using a semi-endogenous growth framework.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Strategy & forecasting + Evals
Source LessWrong · 21
Date Past year
1 item
1 item
"Holy shit" graphs and how to get more of them
Argues that labs should publish auditable capability and deployment metrics, such as internal spending, revenue and AI R&D automation trends, so policymakers can track frontier risks like loss of control and gradual disempowerment.
1 item
AI by Default
A speculative forecast of AI progress before 2028, covering warning shots, gated frontier models, evaluation awareness, concealment by a model, and policy responses to these risks.
1 item
Robustness: Backfire Risks in AI Safety and Governance
Catalogues fifteen mechanisms by which AI safety interventions, such as chain-of-thought monitoring, eval gaming, and commitments, can backfire, mapped against the actors who adapt to them.
1 item
How Would We Know? Reflections on trying to make AI go well amid deep uncertainty
A macrostrategy essay on AI safety under deep uncertainty, arguing for robust strategies, verification capacity and actionable epistemics, with estimates on alignment scaling, control failure, warning shots and governance regimes.
1 item
The Most Important City in AI Safety May Be Singapore
Argues Singapore is a strategic hub for AI safety field-building, international US-China coordination, and verification and evals work, and lists local organisations and next steps.
2 items
A layman-friendly summary of alignment research and its difficulties
A policy-oriented overview arguing AI safety receives tiny resources relative to capabilities and that current alignment techniques, evaluations and guaranteed-safe approaches fall short for AGI/ASI.
Does progress in AI safety require progress in AI capabilities?
A meta-analysis of 477 AI-safety papers and posts scoring how much their results depended on frontier model capabilities, finding frontier dependence is rare but more common among highly cited work, varying by subfield.
1 item
Why autonomous replicating agents are probably not an existential risk (on the contrary)
Argues that near-term autonomous replicating AI agents are unlikely to pose existential risk because frontier labs hold compute and economic advantages, and that they may aid preparedness, though unchecked deployment restrictions could backfire.
1 item
Q2.5 2026 Timelines Update: Uplift and Revenue
AI Futures Project's Q2.5 2026 update revises AI timelines using coding uplift and revenue anchors, finds forecasts converge, and re-grades AI 2027 predictions as reality tracking about 70-90% of its pace.
1 item
Did the alignment community underestimate its power?
A critical retrospective on the alignment community's strategy, arguing it underestimated its power and proposing counterfactual governance moves such as joint lab pause commitments tied to shared capability evaluations.
1 item
On Open-Weights Mythos Capabilities
Argues open-weights models will likely reach frontier cyber capabilities within 24 months, threatening finance and healthcare, and proposes societal and individual preparations such as airgapping mandates.
1 item
5 "Plan A" scenarios
Brainstorms governance options for advanced AI development, including chip-level monitoring, joint projects, deterrence, capability red lines, and liability regimes, and how they could combine into a stronger strategy than Plan A.
1 item
Measuring Is Not Enough Anymore
Argues that METR's time horizon graph, a general capability measurement, has mostly accelerated AI capabilities and investment, so AI safety work should favor opinionated interventions over measuring capabilities.
1 item
Third-parties should focus on scrutinising system cards
Argues that third parties should scrutinise frontier labs' system cards, since card quality will likely degrade as AI automates their writing and labs face stronger incentives to mislead, and proposes concrete recommendations.
1 item
How might outsiders make things go well?
Strategic essay arguing outsiders outside frontier labs are key to AI going well, covering epistemic checks, risk mitigation for reasonable and unreasonable labs, and how outsiders can pressure labs via evals and risk reports.
1 item
Quantitative AI risk assessment: a starting point
Proposes a quantitative probabilistic risk-modelling methodology for AI-enabled cyber attacks, building nine Bayesian models using benchmarks (Cybench, BountyBench) and expert elicitation to estimate AI uplift and harm.
1 item
Monday AI Radar #24
A weekly AI newsletter roundup covering political salience of AI safety, automated AI R&D forecasts, Anthropic's sycophancy reduction, GPT-5.5 cyber risk, and White House model vetting.
2 items
Concrete projects to prepare for superintelligence
Lists concrete projects for a safer transition to superintelligence, including AI character evaluation, sabotage and backdoor audits, deals with misaligned models, coordination tools, space governance, and an ML researcher coalition.
AI's capability improvements haven't come from it getting less affordable
Analyzes METR time-horizon data to argue that inference costs relative to human labor have not risen with frontier capability, so cost is not a bottleneck to automation and timelines are unaffected.
1 item
Building Technology to Drive AI Governance
Argues that technical builders should create measurement, cheap evaluation, and privacy-preserving audit tools to drive AI governance, drawing lessons from climate, food safety, and pandemic response.