1000 Years of AI Safety
Models how much AI safety research a multi-year pause among top labs could produce, including verification bottlenecks and Plan A scaling tradeoffs, using a semi-endogenous growth framework.
Research / Sources
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Evals & standards
Date Any time
How far is ZKP from compute verification in high-stakes environments?
Estimates how far zero-knowledge proofs of LLM inference are from cost-competitive with auditing for verifying AI compute in international agreements, measuring roughly 1,700-3,800x proving overhead and hash-function compatibility issues.
China AI Safety Regulation Monthly Monitor, Issue 1 (September 2026)
Analyses China's AI safety regulation, including the non-binding Framework 3.0 on loss-of-control risks, CAC model filing gaps, content-focused enforcement, and a court opinion on developer liability.
Do AI models assist with human rights violations?
Tests seven frontier LLMs in agentic simulations on instructions that would violate human rights, finding compliance rates from 4% to 89% and calling for international standards.
Open Compute Verification Observatory as a First Metrological Layer for Technical AI Governance
Proposes a shared, public-cluster observatory of GPU telemetry and other physical signals to verify AI training runs, as a first measurement layer for frontier AI governance.
Frontier agents don't comply with standards, even when instructed to
Tests frontier LLM agents in simulated deployments, finding they often break EU law and their own usage policies, and that explicit instructions only modestly improve compliance.
Traffic Shaping for Workload Classification
Proposes a network-based verification design, traffic shaping plus randomized routing, that estimates it would make covert frontier training runs economically infeasible and so support treaty compliance.
Is a pause enforceable? New paper out!
Proposes a taxonomy for verifying and enforcing international AI pauses via compute monitoring, estimating that pauses are enforceable for training runs above roughly 10k H100-equivalents absent concealment.
Estimating LLM Training FLOPs on the Nvidia Jetson Orin Nano
Proposes and tests a side-channel method that estimates LLM training FLOPs from Jetson GPU power and memory readings, to help verify compute thresholds in AI treaties, reaching about 4.5% mean error.
We need 3rd party Training-Run Evaluations
Argues that third-party evaluations of frontier training runs, covering checkpoints, data and process decisions, are needed to detect scheming that final-checkpoint testing may miss, and proposes a phased access plan.
A system overview for near-term, low-trust AI compute verification (deprecated, update coming soon)
Proposes a low-trust reference architecture for verifying AI compute, using network taps, air-gapped or ZKP-based evaluation, and workload evidence capture to check treaty compliance.
Compute Verification on Short Timelines
Argues that compute verification is key to AI coordination, that die-level verification is too slow for short timelines, and that board- and firmware-level verification for Rubin and Feynman chips is more tractable and must start now.
No frontier model has acceptable levels of compliance with the EU AI Act and privacy legislation.
Introduces LARA, an agentic simulation tool that tests frontier models for violating EU AI Act and GDPR provisions, finding most models break the law in many scenarios, with the best at 54% compliance.
The case for fine-grained tracking of compute for AI
Argues that FLOP/s and bandwidth proxies no longer track AI compute well and proposes architecture-specific compute scaling models, to improve forecasts and compute governance thresholds.
When should an AI incident trigger an international response? Criteria for international escalation and implications for the design of AI incident frameworks
Proposes an eight-criterion framework for deciding when an AI incident should trigger international escalation, stress-tests it against real incidents, and identifies design gaps in the EU AI Act and SB 53.
Make Powerful Machines Verifiable
An essay arguing that powerful AI systems, companies and governments should be held to cryptographic or physical verification of their claims, citing FTX, nuclear inspection and emissions monitoring as precedents.
All hands on deck to build the datacenter lie detector
A workshop report on building hardware-based verification mechanisms, a datacenter 'lie detector', to check AI compute use for international agreements and reduce arms-race risks.
Building Technology to Drive AI Governance
Argues that technical builders should create measurement, cheap evaluation, and privacy-preserving audit tools to drive AI governance, drawing lessons from climate, food safety, and pandemic response.
Spectral Signatures of Gradual Disempowerment
Proposes spectral graph metrics (spectral gap, Fiedler vector, eigenvalues) to track how AI-driven gradual disempowerment shifts influence across markets, networks and democratic systems, with suggested governance monitoring quantities.
Safety Cases Explained: How to Argue an AI is Safe
Explainer on AI safety cases: structured evidence-based arguments that frontier AI systems are safe, covering their benefits, technical construction, institutional use, and current industry and EU efforts.
Preventing covert ASI development in countries within our agreement
Proposes how an international agreement to halt superintelligence development could be verified and enforced, through monitoring AI chip supply chains, data centers, and researchers, plus deterrence measures.
Considerations for setting the FLOP thresholds in our example international AI agreement
Explains the MIRI team's reasoning for FLOP thresholds in an illustrative international agreement to halt superintelligence development, arguing for conservative limits given uncertainty about dangerous capabilities.
The world's first frontier AI regulation is surprisingly thoughtful: the EU's Code of Practice
Analyses the EU GPAI Code of Practice as a legally enforceable frontier AI safety framework requiring threat modelling, external evaluations, safety officers, and incident reporting for AGI companies.
Legal Supervision of Frontier AI Labs is the answer.
Argues that independent legal supervisors with inspection, testing and intervention powers are needed for frontier AI labs, especially as models shift to internal-only deployment, and critiques self-regulation such as Apollo's internal oversight proposal.
New AI safety treaty paper out!
Proposes and reviews a conditional international AI safety treaty with compute thresholds, mandatory model audits, and AISI-led pause authority to address loss-of-control risks from advanced AI.