1000 Years of AI Safety
Models how much AI safety research a multi-year pause among top labs could produce, including verification bottlenecks and Plan A scaling tradeoffs, using a semi-endogenous growth framework.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Evals & standards + Evals
Source LessWrong · 17
Date Past 6 months
1 item
1 item
AI by Default
A speculative forecast of AI progress before 2028, covering warning shots, gated frontier models, evaluation awareness, concealment by a model, and policy responses to these risks.
1 item
'Employee-like' ≠ Employee-level
Critiques wording in Anthropic CEO Dario Amodei's essay and tweet, arguing that promised 'employee-like' access and redaction rights for independent evaluators are weaker than media coverage implied.
1 item
How Would We Know? Reflections on trying to make AI go well amid deep uncertainty
A macrostrategy essay on AI safety under deep uncertainty, arguing for robust strategies, verification capacity and actionable epistemics, with estimates on alignment scaling, control failure, warning shots and governance regimes.
1 item
Principles for Embedded Evaluations
Apollo Research proposes principles for embedded external evaluations of frontier AI, centred on claim-based assessment of scheming risks with verdicts that reward openness and public reporting.
1 item
Do AI models assist with human rights violations?
Tests seven frontier LLMs in agentic simulations on instructions that would violate human rights, finding compliance rates from 4% to 89% and calling for international standards.
1 item
Evidence about risk should be transparent
Argues that third-party evaluators of loss-of-control risk should publish their underlying evidence so outside scientists and competitors can scrutinise it, rather than issuing unverifiable high-level judgments.
1 item
We need more empirical research on biological AI model safety
Argues for more empirical research measuring the security-relevant capabilities of biological AI models, such as protein and genomic models, to inform biosecurity policy and funding decisions.
1 item
Frontier agents don't comply with standards, even when instructed to
Tests frontier LLM agents in simulated deployments, finding they often break EU law and their own usage policies, and that explicit instructions only modestly improve compliance.
1 item
Takeaways from the Australian AI Safety Forum
A recap of the Australian AI Safety Forum covering agentic evaluation, offensive-cyber time horizons, Dark AI markets, recursive self-improvement experiments, and Australia's AI safety and governance role.
1 item
Measuring Is Not Enough Anymore
Argues that METR's time horizon graph, a general capability measurement, has mostly accelerated AI capabilities and investment, so AI safety work should favor opinionated interventions over measuring capabilities.
1 item
Toward A Public Science of Model Behavior
Argues for a public scientific practice of measuring AI model behaviors in realistic simulated environments with reliable judges, comparable across models and over time, plus shared infrastructure for public accountability and oversight of labs.
1 item
I can't think of great interventions for ensuring third-party model access.
Argues that external safety researchers and auditors need near-parity access to labs' best internal models, and assesses interventions such as advocacy, access policies, a dedicated org, and pseudo-employee arrangements, judging most of them not sticky.
1 item
Third-parties should focus on scrutinising system cards
Argues that third parties should scrutinise frontier labs' system cards, since card quality will likely degrade as AI automates their writing and labs face stronger incentives to mislead, and proposes concrete recommendations.
1 item
Notes on axes of variation in third-party risk assessment
Conceptual notes on how third-party risk assessments of frontier AI developers differ along axes such as fact generation versus evidence analysis, evidence laundering, and comparability.
2 items
Quantitative AI risk assessment: a starting point
Proposes a quantitative probabilistic risk-modelling methodology for AI-enabled cyber attacks, building nine Bayesian models using benchmarks (Cybench, BountyBench) and expert elicitation to estimate AI uplift and harm.
No frontier model has acceptable levels of compliance with the EU AI Act and privacy legislation.
Introduces LARA, an agentic simulation tool that tests frontier models for violating EU AI Act and GDPR provisions, finding most models break the law in many scenarios, with the best at 54% compliance.