1000 Years of AI Safety
Models how much AI safety research a multi-year pause among top labs could produce, including verification bottlenecks and Plan A scaling tradeoffs, using a semi-endogenous growth framework.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Evals & standards + Evals
Source LessWrong · 37
Date Any time
1 item
1 item
AI by Default
A speculative forecast of AI progress before 2028, covering warning shots, gated frontier models, evaluation awareness, concealment by a model, and policy responses to these risks.
1 item
'Employee-like' ≠ Employee-level
Critiques wording in Anthropic CEO Dario Amodei's essay and tweet, arguing that promised 'employee-like' access and redaction rights for independent evaluators are weaker than media coverage implied.
1 item
How Would We Know? Reflections on trying to make AI go well amid deep uncertainty
A macrostrategy essay on AI safety under deep uncertainty, arguing for robust strategies, verification capacity and actionable epistemics, with estimates on alignment scaling, control failure, warning shots and governance regimes.
1 item
Principles for Embedded Evaluations
Apollo Research proposes principles for embedded external evaluations of frontier AI, centred on claim-based assessment of scheming risks with verdicts that reward openness and public reporting.
1 item
Do AI models assist with human rights violations?
Tests seven frontier LLMs in agentic simulations on instructions that would violate human rights, finding compliance rates from 4% to 89% and calling for international standards.
1 item
Evidence about risk should be transparent
Argues that third-party evaluators of loss-of-control risk should publish their underlying evidence so outside scientists and competitors can scrutinise it, rather than issuing unverifiable high-level judgments.
1 item
We need more empirical research on biological AI model safety
Argues for more empirical research measuring the security-relevant capabilities of biological AI models, such as protein and genomic models, to inform biosecurity policy and funding decisions.
1 item
Frontier agents don't comply with standards, even when instructed to
Tests frontier LLM agents in simulated deployments, finding they often break EU law and their own usage policies, and that explicit instructions only modestly improve compliance.
1 item
Takeaways from the Australian AI Safety Forum
A recap of the Australian AI Safety Forum covering agentic evaluation, offensive-cyber time horizons, Dark AI markets, recursive self-improvement experiments, and Australia's AI safety and governance role.
1 item
Measuring Is Not Enough Anymore
Argues that METR's time horizon graph, a general capability measurement, has mostly accelerated AI capabilities and investment, so AI safety work should favor opinionated interventions over measuring capabilities.
1 item
Toward A Public Science of Model Behavior
Argues for a public scientific practice of measuring AI model behaviors in realistic simulated environments with reliable judges, comparable across models and over time, plus shared infrastructure for public accountability and oversight of labs.
1 item
I can't think of great interventions for ensuring third-party model access.
Argues that external safety researchers and auditors need near-parity access to labs' best internal models, and assesses interventions such as advocacy, access policies, a dedicated org, and pseudo-employee arrangements, judging most of them not sticky.
1 item
Third-parties should focus on scrutinising system cards
Argues that third parties should scrutinise frontier labs' system cards, since card quality will likely degrade as AI automates their writing and labs face stronger incentives to mislead, and proposes concrete recommendations.
1 item
Notes on axes of variation in third-party risk assessment
Conceptual notes on how third-party risk assessments of frontier AI developers differ along axes such as fact generation versus evidence analysis, evidence laundering, and comparability.
2 items
Quantitative AI risk assessment: a starting point
Proposes a quantitative probabilistic risk-modelling methodology for AI-enabled cyber attacks, building nine Bayesian models using benchmarks (Cybench, BountyBench) and expert elicitation to estimate AI uplift and harm.
No frontier model has acceptable levels of compliance with the EU AI Act and privacy legislation.
Introduces LARA, an agentic simulation tool that tests frontier models for violating EU AI Act and GDPR provisions, finding most models break the law in many scenarios, with the best at 54% compliance.
1 item
Toward a Better Evaluations Ecosystem
Argues that frontier model benchmark numbers are inconsistent across labs and proposes standardized third-party auditing of evaluations used in releases and safety decisions.
1 item
OpenAI's red line for AI self-improvement is fundamentally flawed
Argues OpenAI's Critical self-improvement threshold in its Preparedness Framework fires too late, is unmeasurable and self-certified, and proposes an independently evaluated METR time-horizon trigger.
1 item
We're actually running out of benchmarks to upper bound AI capabilities
Argues that fixed benchmarks can no longer upper-bound frontier models' dangerous capabilities, and surveys alternatives: uplift studies, expert forecasting, and third-party risk audits.
1 item
Anthropic Responsible Scaling Policy v3: Dive Into The Details
Critiques Anthropic's Responsible Scaling Policy v3, arguing its shift to flexible 'strong argument' standards, weaker release gates and vague commitments makes it rely on trust in Anthropic.
1 item
Concrete projects to prepare for superintelligence
Lists concrete projects for a safer transition to superintelligence, including AI character evaluation, sabotage and backdoor audits, deals with misaligned models, coordination tools, space governance, and an ML researcher coalition.
1 item
Why AI Evaluation Regimes are bad
An opinion piece arguing that third-party AI evaluation organisations are harmful: their theory of change depends on nonexistent regulation, they shift the burden of proof onto the public, and they are not independent of AI companies. It calls for banning dangerous AI capabilities instead.
1 item
Responsible Scaling Policy v3
A detailed take on Anthropic's Responsible Scaling Policy v3, covering the move away from hard commitments, the original RSP goals, and lessons from ASL-3 safeguards and security.
1 item
Building Technology to Drive AI Governance
Argues that technical builders should create measurement, cheap evaluation, and privacy-preserving audit tools to drive AI governance, drawing lessons from climate, food safety, and pandemic response.
1 item
Safety Cases Explained: How to Argue an AI is Safe
Explainer on AI safety cases: structured evidence-based arguments that frontier AI systems are safe, covering their benefits, technical construction, institutional use, and current industry and EU efforts.
1 item
Summary and Comments on Anthropic's Pilot Sabotage Risk Report
Summarises Anthropic's pilot safety case for Claude Opus 4 and 4.1 on sabotage risk, its METR review, and the evidence and weaknesses in the argument.
1 item
All the labs AI safety plans: 2025 edition
Compares the 2025 safety frameworks of Anthropic, Google DeepMind and OpenAI, covering their dangerous-capability thresholds, evaluation procedures, deployment and security safeguards, and governance structures.
1 item
AI Companion Piece
Commentary on a study finding AI persuasion grows with post-training and scale, on teen AI companion use, and on how personalization could undermine safety evaluations and raise persuasion and engagement risks.
1 item
New website analyzing AI companies' model evals
A beta website critiques frontier labs' model cards for dangerous-capability evals, arguing that weak elicitation, unclear thresholds and missing human baselines make the safety claims unsupported.
1 item
OpenAI Preparedness Framework 2.0
Critiques OpenAI's Preparedness Framework 2.0, arguing it drops persuasion and autonomous replication as tracked risks, relies on defense-in-depth safeguards, and sets high-risk thresholds too late.
1 item
Existing Safety Frameworks Imply Unreasonable Confidence
Argues that frontier labs' public safety frameworks (Anthropic, OpenAI, DeepMind, Meta, xAI) lack proper risk-management practice, leaving unknown risks unaddressed and projecting unwarranted confidence in their safety.}
2 items
AI companies should be safety-testing the most capable versions of their models
Argues that AI companies should safety-test task-specific fine-tuned versions of their models to find worst-case dangerous capabilities, and assesses whether OpenAI follows through on this.
New AI safety treaty paper out!
Proposes and reviews a conditional international AI safety treaty with compute thresholds, mandatory model audits, and AISI-led pause authority to address loss-of-control risks from advanced AI.
1 item
Anthropic: Progress from our Frontier Red Team
Anthropic's Frontier Red Team reports that Claude models are showing early warning signs of dual-use cyber and biology capability gains, short of national-security risk thresholds, and describes government evaluation partnerships.
1 item
Some lessons from the OpenAI-FrontierMath debacle
Argues that undisclosed OpenAI funding and exclusive data access for the FrontierMath benchmark undermined its transparency, and proposes disclosure and written data-use rules for future AI evaluations.
1 item
Improving Our Safety Cases Using Upper and Lower Bounds
Proposes framing AI safety cases as upper and lower bounds on sufficient safety measures, separating whether a measure would suffice from whether it is technically implementable, with examples like interpretability, sandbox escape tests, and catastrophic action detection.