1000 Years of AI Safety
Models how much AI safety research a multi-year pause among top labs could produce, including verification bottlenecks and Plan A scaling tradeoffs, using a semi-endogenous growth framework.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Strategy & forecasting + Evals
Source LessWrong · 31
Date Any time
1 item
1 item
"Holy shit" graphs and how to get more of them
Argues that labs should publish auditable capability and deployment metrics, such as internal spending, revenue and AI R&D automation trends, so policymakers can track frontier risks like loss of control and gradual disempowerment.
1 item
AI by Default
A speculative forecast of AI progress before 2028, covering warning shots, gated frontier models, evaluation awareness, concealment by a model, and policy responses to these risks.
1 item
Robustness: Backfire Risks in AI Safety and Governance
Catalogues fifteen mechanisms by which AI safety interventions, such as chain-of-thought monitoring, eval gaming, and commitments, can backfire, mapped against the actors who adapt to them.
1 item
How Would We Know? Reflections on trying to make AI go well amid deep uncertainty
A macrostrategy essay on AI safety under deep uncertainty, arguing for robust strategies, verification capacity and actionable epistemics, with estimates on alignment scaling, control failure, warning shots and governance regimes.
1 item
The Most Important City in AI Safety May Be Singapore
Argues Singapore is a strategic hub for AI safety field-building, international US-China coordination, and verification and evals work, and lists local organisations and next steps.
2 items
A layman-friendly summary of alignment research and its difficulties
A policy-oriented overview arguing AI safety receives tiny resources relative to capabilities and that current alignment techniques, evaluations and guaranteed-safe approaches fall short for AGI/ASI.
Does progress in AI safety require progress in AI capabilities?
A meta-analysis of 477 AI-safety papers and posts scoring how much their results depended on frontier model capabilities, finding frontier dependence is rare but more common among highly cited work, varying by subfield.
1 item
Why autonomous replicating agents are probably not an existential risk (on the contrary)
Argues that near-term autonomous replicating AI agents are unlikely to pose existential risk because frontier labs hold compute and economic advantages, and that they may aid preparedness, though unchecked deployment restrictions could backfire.
1 item
Q2.5 2026 Timelines Update: Uplift and Revenue
AI Futures Project's Q2.5 2026 update revises AI timelines using coding uplift and revenue anchors, finds forecasts converge, and re-grades AI 2027 predictions as reality tracking about 70-90% of its pace.
1 item
Did the alignment community underestimate its power?
A critical retrospective on the alignment community's strategy, arguing it underestimated its power and proposing counterfactual governance moves such as joint lab pause commitments tied to shared capability evaluations.
1 item
On Open-Weights Mythos Capabilities
Argues open-weights models will likely reach frontier cyber capabilities within 24 months, threatening finance and healthcare, and proposes societal and individual preparations such as airgapping mandates.
1 item
5 "Plan A" scenarios
Brainstorms governance options for advanced AI development, including chip-level monitoring, joint projects, deterrence, capability red lines, and liability regimes, and how they could combine into a stronger strategy than Plan A.
1 item
Measuring Is Not Enough Anymore
Argues that METR's time horizon graph, a general capability measurement, has mostly accelerated AI capabilities and investment, so AI safety work should favor opinionated interventions over measuring capabilities.
1 item
Third-parties should focus on scrutinising system cards
Argues that third parties should scrutinise frontier labs' system cards, since card quality will likely degrade as AI automates their writing and labs face stronger incentives to mislead, and proposes concrete recommendations.
1 item
How might outsiders make things go well?
Strategic essay arguing outsiders outside frontier labs are key to AI going well, covering epistemic checks, risk mitigation for reasonable and unreasonable labs, and how outsiders can pressure labs via evals and risk reports.
1 item
Quantitative AI risk assessment: a starting point
Proposes a quantitative probabilistic risk-modelling methodology for AI-enabled cyber attacks, building nine Bayesian models using benchmarks (Cybench, BountyBench) and expert elicitation to estimate AI uplift and harm.
1 item
Monday AI Radar #24
A weekly AI newsletter roundup covering political salience of AI safety, automated AI R&D forecasts, Anthropic's sycophancy reduction, GPT-5.5 cyber risk, and White House model vetting.
2 items
Concrete projects to prepare for superintelligence
Lists concrete projects for a safer transition to superintelligence, including AI character evaluation, sabotage and backdoor audits, deals with misaligned models, coordination tools, space governance, and an ML researcher coalition.
AI's capability improvements haven't come from it getting less affordable
Analyzes METR time-horizon data to argue that inference costs relative to human labor have not risen with frontier capability, so cost is not a bottleneck to automation and timelines are unaffected.
1 item
Building Technology to Drive AI Governance
Argues that technical builders should create measurement, cheap evaluation, and privacy-preserving audit tools to drive AI governance, drawing lessons from climate, food safety, and pandemic response.
1 item
What's up with Anthropic predicting AGI by early 2027?
Critiques Anthropic's prediction of powerful AI by early 2027, operationalizes it into falsifiable AI R&D automation milestones, and argues it is unlikely given current capability trends.
1 item
Asking (Some Of) The Right Questions
A commentary on a statement against building superintelligence soon, laying out which developments would shift the author's estimates of existential risk, alignment difficulty, timelines, and the case for pausing.
1 item
AI and Cheap Weapons
Argues that falling costs of weapons-capable AI will cause proliferation of mass-destruction capabilities, surveying offensive uses in bio, chemical, cyber, drone and mirror-life weapons and superintelligence risks, with policy to follow.
1 item
[Retracted] Guess I Was Wrong About AIxBio Risks
A retracted LessWrong post reviews a study where genome language models generated functional phages, and argues DNA synthesis screening should use functional annotation to catch AI-enabled biothreats.
1 item
We are likely in an AI overhang, and this is bad.
Argues that untapped capabilities in current models mean safety frameworks like red lines and RSPs may be too late, since overhangs are hidden and people underestimate them.
1 item
Shanzson AI 2027 Timeline
A speculative alternative AI 2027 timeline in which misaligned AI incidents trigger strict regulation and bans, and argues that AI companies may be tempted to build bioweapons.
1 item
Analyzing A Critique Of The AI 2027 Timeline Forecasts
A long commentary defending the AI 2027 timeline forecasts against a critique, analyzing METR time-horizon extrapolation, superexponential assumptions, and the critique's tone; it is about AI forecasting methodology rather than safety directly.
1 item
Thoughts on AI 2027
A MIRI commentary on the AI 2027 forecast argues timelines are plausibly shorter than the median, stresses modal versus median timing, and gives section-by-section critiques of its scenario.
1 item
AI 2027: Responses
A roundup of reactions to the AI 2027 scenario forecast of superhuman AI and recursive self-improvement, covering its risk implications, critiques of self-fulfilling prophecy, and forecast track records.
1 item
The Takeoff Speeds Model Predicts We May Be Entering Crunch Time
Updates the Takeoff Speeds model with recent compute trends and R&D automation estimates, predicting full economic automation around 2030 and a crunch-time window for AI risk policy planning.