1000 Years of AI Safety
Models how much AI safety research a multi-year pause among top labs could produce, including verification bottlenecks and Plan A scaling tradeoffs, using a semi-endogenous growth framework.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Evals + Governance
Source LessWrong · 68
Date Any time
1 item
1 item
"Holy shit" graphs and how to get more of them
Argues that labs should publish auditable capability and deployment metrics, such as internal spending, revenue and AI R&D automation trends, so policymakers can track frontier risks like loss of control and gradual disempowerment.
1 item
AI by Default
A speculative forecast of AI progress before 2028, covering warning shots, gated frontier models, evaluation awareness, concealment by a model, and policy responses to these risks.
1 item
Robustness: Backfire Risks in AI Safety and Governance
Catalogues fifteen mechanisms by which AI safety interventions, such as chain-of-thought monitoring, eval gaming, and commitments, can backfire, mapped against the actors who adapt to them.
1 item
'Employee-like' ≠ Employee-level
Critiques wording in Anthropic CEO Dario Amodei's essay and tweet, arguing that promised 'employee-like' access and redaction rights for independent evaluators are weaker than media coverage implied.
1 item
How Would We Know? Reflections on trying to make AI go well amid deep uncertainty
A macrostrategy essay on AI safety under deep uncertainty, arguing for robust strategies, verification capacity and actionable epistemics, with estimates on alignment scaling, control failure, warning shots and governance regimes.
1 item
Principles for Embedded Evaluations
Apollo Research proposes principles for embedded external evaluations of frontier AI, centred on claim-based assessment of scheming risks with verdicts that reward openness and public reporting.
1 item
Do AI models assist with human rights violations?
Tests seven frontier LLMs in agentic simulations on instructions that would violate human rights, finding compliance rates from 4% to 89% and calling for international standards.
1 item
Evidence about risk should be transparent
Argues that third-party evaluators of loss-of-control risk should publish their underlying evidence so outside scientists and competitors can scrutinise it, rather than issuing unverifiable high-level judgments.
1 item
The Most Important City in AI Safety May Be Singapore
Argues Singapore is a strategic hub for AI safety field-building, international US-China coordination, and verification and evals work, and lists local organisations and next steps.
2 items
A layman-friendly summary of alignment research and its difficulties
A policy-oriented overview arguing AI safety receives tiny resources relative to capabilities and that current alignment techniques, evaluations and guaranteed-safe approaches fall short for AGI/ASI.
Does progress in AI safety require progress in AI capabilities?
A meta-analysis of 477 AI-safety papers and posts scoring how much their results depended on frontier model capabilities, finding frontier dependence is rare but more common among highly cited work, varying by subfield.
1 item
Why autonomous replicating agents are probably not an existential risk (on the contrary)
Argues that near-term autonomous replicating AI agents are unlikely to pose existential risk because frontier labs hold compute and economic advantages, and that they may aid preparedness, though unchecked deployment restrictions could backfire.
1 item
We need more empirical research on biological AI model safety
Argues for more empirical research measuring the security-relevant capabilities of biological AI models, such as protein and genomic models, to inform biosecurity policy and funding decisions.
1 item
Q2.5 2026 Timelines Update: Uplift and Revenue
AI Futures Project's Q2.5 2026 update revises AI timelines using coding uplift and revenue anchors, finds forecasts converge, and re-grades AI 2027 predictions as reality tracking about 70-90% of its pace.
1 item
Frontier agents don't comply with standards, even when instructed to
Tests frontier LLM agents in simulated deployments, finding they often break EU law and their own usage policies, and that explicit instructions only modestly improve compliance.
1 item
Did the alignment community underestimate its power?
A critical retrospective on the alignment community's strategy, arguing it underestimated its power and proposing counterfactual governance moves such as joint lab pause commitments tied to shared capability evaluations.
1 item
On Open-Weights Mythos Capabilities
Argues open-weights models will likely reach frontier cyber capabilities within 24 months, threatening finance and healthcare, and proposes societal and individual preparations such as airgapping mandates.
1 item
AI Rights Aren't Safety-Neutral: A Quick Follow-Up to the Consciousness Cluster
Prompting or fine-tuning models to claim or deny legal rights shifts power-seeking and corrigibility benchmarks, suggesting AI rights framing is not safety-neutral and should inform legislation.
1 item
AI Voice Phishing Performs on Par With Human Scammers at a Fraction of the Cost
A 4,100-person study finds AI voice models match or beat human scammers in eliciting compliance, and that AI vishing is already economically profitable, so the risk lies in automation.
1 item
Takeaways from the Australian AI Safety Forum
A recap of the Australian AI Safety Forum covering agentic evaluation, offensive-cyber time horizons, Dark AI markets, recursive self-improvement experiments, and Australia's AI safety and governance role.
1 item
5 "Plan A" scenarios
Brainstorms governance options for advanced AI development, including chip-level monitoring, joint projects, deterrence, capability red lines, and liability regimes, and how they could combine into a stronger strategy than Plan A.
1 item
Measuring Is Not Enough Anymore
Argues that METR's time horizon graph, a general capability measurement, has mostly accelerated AI capabilities and investment, so AI safety work should favor opinionated interventions over measuring capabilities.
1 item
Toward A Public Science of Model Behavior
Argues for a public scientific practice of measuring AI model behaviors in realistic simulated environments with reliable judges, comparable across models and over time, plus shared infrastructure for public accountability and oversight of labs.
1 item
I can't think of great interventions for ensuring third-party model access.
Argues that external safety researchers and auditors need near-parity access to labs' best internal models, and assesses interventions such as advocacy, access policies, a dedicated org, and pseudo-employee arrangements, judging most of them not sticky.
1 item
Third-parties should focus on scrutinising system cards
Argues that third parties should scrutinise frontier labs' system cards, since card quality will likely degrade as AI automates their writing and labs face stronger incentives to mislead, and proposes concrete recommendations.
1 item
How might outsiders make things go well?
Strategic essay arguing outsiders outside frontier labs are key to AI going well, covering epistemic checks, risk mitigation for reasonable and unreasonable labs, and how outsiders can pressure labs via evals and risk reports.
1 item
Notes on axes of variation in third-party risk assessment
Conceptual notes on how third-party risk assessments of frontier AI developers differ along axes such as fact generation versus evidence analysis, evidence laundering, and comparability.
2 items
Quantitative AI risk assessment: a starting point
Proposes a quantitative probabilistic risk-modelling methodology for AI-enabled cyber attacks, building nine Bayesian models using benchmarks (Cybench, BountyBench) and expert elicitation to estimate AI uplift and harm.
No frontier model has acceptable levels of compliance with the EU AI Act and privacy legislation.
Introduces LARA, an agentic simulation tool that tests frontier models for violating EU AI Act and GDPR provisions, finding most models break the law in many scenarios, with the best at 54% compliance.
1 item
How should we update on AI-enabled coups post-Mythos?
Argues that Anthropic's Claude Mythos and its Project Glasswing access controls make certain AI-enabled coup pathways more plausible, and that private lab control over access to dangerous cyber capabilities is a governance concern.
1 item
Monday AI Radar #24
A weekly AI newsletter roundup covering political salience of AI safety, automated AI R&D forecasts, Anthropic's sycophancy reduction, GPT-5.5 cyber risk, and White House model vetting.
1 item
Toward a Better Evaluations Ecosystem
Argues that frontier model benchmark numbers are inconsistent across labs and proposes standardized third-party auditing of evaluations used in releases and safety decisions.
1 item
OpenAI's red line for AI self-improvement is fundamentally flawed
Argues OpenAI's Critical self-improvement threshold in its Preparedness Framework fires too late, is unmeasurable and self-certified, and proposes an independently evaluated METR time-horizon trigger.
1 item
We're actually running out of benchmarks to upper bound AI capabilities
Argues that fixed benchmarks can no longer upper-bound frontier models' dangerous capabilities, and surveys alternatives: uplift studies, expert forecasting, and third-party risk audits.
1 item
Anthropic Responsible Scaling Policy v3: Dive Into The Details
Critiques Anthropic's Responsible Scaling Policy v3, arguing its shift to flexible 'strong argument' standards, weaker release gates and vague commitments makes it rely on trust in Anthropic.
2 items
Concrete projects to prepare for superintelligence
Lists concrete projects for a safer transition to superintelligence, including AI character evaluation, sabotage and backdoor audits, deals with misaligned models, coordination tools, space governance, and an ML researcher coalition.
AI's capability improvements haven't come from it getting less affordable
Analyzes METR time-horizon data to argue that inference costs relative to human labor have not risen with frontier capability, so cost is not a bottleneck to automation and timelines are unaffected.
1 item
Why AI Evaluation Regimes are bad
An opinion piece arguing that third-party AI evaluation organisations are harmful: their theory of change depends on nonexistent regulation, they shift the burden of proof onto the public, and they are not independent of AI companies. It calls for banning dangerous AI capabilities instead.
1 item
Negligent AI: Reasonable Care for AI Safety
Proposes negligence law's reasonable-care standard as an alignment target and tests whether frontier models give less permissive advice when asked to apply negligence law.