We urgently need to practice before we can pause
Argues a global AI pause needs incremental precedent, with third-party verification of companies' safety-motivated slowdowns building trust and assessor competence before a pause.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Evals & standards
Source LessWrong · 100
Date Any time
2 items
Argue about Risk Management Speed, not p(doom)
Proposes a dynamic 7-level scale for frontier AI labs' risk management culture, arguing that the rate of safety improvement, not a static p(doom), should drive safety forecasts and debates.
2 items
Which AI economics and governance research most needs independent expert evaluation? (help The Unjournal prioritize it)
The Unjournal asks for input on prioritizing AI economics and governance research for public expert evaluation, covering growth forecasts, recursive self-improvement models, compute smuggling estimates, and AI liability work.
1000 Years of AI Safety
Models how much AI safety research a multi-year pause among top labs could produce, including verification bottlenecks and Plan A scaling tradeoffs, using a semi-endogenous growth framework.
1 item
What do the other 80 percent mean?
An Austrian author argues that public AI catastrophe estimates lack a breakdown of outcomes, cites prior work on decomposing AI risk, and proposes analog fallback infrastructure as preparation.
2 items
AI by Default
A speculative forecast of AI progress before 2028, covering warning shots, gated frontier models, evaluation awareness, concealment by a model, and policy responses to these risks.
How far is ZKP from compute verification in high-stakes environments?
Estimates how far zero-knowledge proofs of LLM inference are from cost-competitive with auditing for verifying AI compute in international agreements, measuring roughly 1,700-3,800x proving overhead and hash-function compatibility issues.
1 item
China AI Safety Regulation Monthly Monitor, Issue 1 (September 2026)
Analyses China's AI safety regulation, including the non-binding Framework 3.0 on loss-of-control risks, CAC model filing gaps, content-focused enforcement, and a court opinion on developer liability.
1 item
'Employee-like' ≠ Employee-level
Critiques wording in Anthropic CEO Dario Amodei's essay and tweet, arguing that promised 'employee-like' access and redaction rights for independent evaluators are weaker than media coverage implied.
1 item
Proof of useful work for verifying AI treaties
Literature review of proof of useful work, a cryptographic method for GPUs to prove compute use, as a way to verify international AI pause treaties, finding open gaps in guarantees and hardware assumptions.
1 item
Towards embedded evaluations for scheming propensities
Proposes four claims a scheming safety case must substantiate and argues that third-party embedded evaluators need persistent, deep access to training, monitoring and chain-of-thought to verify them.
1 item
How Would We Know? Reflections on trying to make AI go well amid deep uncertainty
A macrostrategy essay on AI safety under deep uncertainty, arguing for robust strategies, verification capacity and actionable epistemics, with estimates on alignment scaling, control failure, warning shots and governance regimes.
1 item
Principles for Embedded Evaluations
Apollo Research proposes principles for embedded external evaluations of frontier AI, centred on claim-based assessment of scheming risks with verdicts that reward openness and public reporting.
1 item
Do AI models assist with human rights violations?
Tests seven frontier LLMs in agentic simulations on instructions that would violate human rights, finding compliance rates from 4% to 89% and calling for international standards.
1 item
Evidence about risk should be transparent
Argues that third-party evaluators of loss-of-control risk should publish their underlying evidence so outside scientists and competitors can scrutinise it, rather than issuing unverifiable high-level judgments.
1 item
Weight smuggling likely defeats attempts to cap FLOPs per training run
Argues that per-run FLOP caps for frontier training can be covertly bypassed by smuggling weights into new runs or aggregating parallel runs, and suggests a global training budget instead.
1 item
If METR is overworked, how to alleviate the bottleneck?
An opinion post proposing new university degrees in AI alignment and 'Intellidynamics' to staff a government-funded independent AI auditing agency that would oversee frontier labs.
1 item
How we might actually pace the frontier: A proposal for AI companies to do public pacing exercises.
Proposes that AI companies run public, independently evaluated pacing exercises such as multi-day training halts, to build verification capacity and prepare for larger frontier slowdowns.
2 items
Categorical taboos are much better than threshold taboos: neuralese edition
Argues that neuralese should be governed by a categorical taboo rather than a numerical threshold, using nuclear and doping analogies, because chain-of-thought monitorability depends on keeping self-communication in natural language.
AI Safety: A Short FAQ for Mathematicians
An FAQ urging mathematicians to work on AI safety, surveying research areas such as agent foundations, embedded agency, and pause and technical governance, plus ways to get involved.
2 items
Fighting scope and frame control from the frontier-labs
Argues that frontier AI labs' safety communications and scope-limited external evaluations warrant adversarial external pressure, and proposes concrete signals and transparency measures that would justify less scepticism.
Recommendations for People Getting into Technical AI Governance Research
MIRI shares reading lists and research advice for newcomers to technical AI governance, drawn from its 2026 fellowship, to help researchers enter the field.
1 item
Open Compute Verification Observatory as a First Metrological Layer for Technical AI Governance
Proposes a shared, public-cluster observatory of GPU telemetry and other physical signals to verify AI training runs, as a first measurement layer for frontier AI governance.
1 item
AI Safety in Japan has deeper problems than capital allocation
An opinion piece arguing that Japan's AI safety efforts lack a clear field strategy compared with Singapore and the Philippines, which have pivoted to AI verification and governance work, and urging Japanese institutes to set priorities.
1 item
We need more empirical research on biological AI model safety
Argues for more empirical research measuring the security-relevant capabilities of biological AI models, such as protein and genomic models, to inform biosecurity policy and funding decisions.
1 item
Frontier agents don't comply with standards, even when instructed to
Tests frontier LLM agents in simulated deployments, finding they often break EU law and their own usage policies, and that explicit instructions only modestly improve compliance.
1 item
How to pace the US frontier
Proposes concrete domestic US policies to pace frontier AI development, including compute allocation minimums for safety research, caps on AI R&D use, and safety-case-based existential risk thresholds enforced by third-party assessors.
1 item
Traffic Shaping for Workload Classification
Proposes a network-based verification design, traffic shaping plus randomized routing, that estimates it would make covert frontier training runs economically infeasible and so support treaty compliance.
1 item
An International AI Slowdown Is Ready Whenever Politicians Are
Argues that an international AI slowdown can be verified with human inspectors given whole-lab access to frontier labs, and that the technical verification barrier is not a reason to delay it.
1 item
Why Formation Research is Working on Secret Loyalties
Formation Research explains its choice to focus on empirical secret-loyalty research, building model organisms of secretly loyal AIs to stress-test detection and mitigation, tied to power-concentration risk.
2 items
My Assessment of Plan A's Compute Verification Strategy (+ open questions)
Analyses technical mechanisms for verifying that datacenter compute is used only for inference, such as interconnect limits, memory wipes, replay, and zero-knowledge proofs, to slow unmonitored frontier training.
Is a pause enforceable? New paper out!
Proposes a taxonomy for verifying and enforcing international AI pauses via compute monitoring, estimating that pauses are enforceable for training runs above roughly 10k H100-equivalents absent concealment.
1 item
Auditor-in-a-Box: Tools for Third-Party Auditing
Proposes running an open-source LLM inside a trusted execution environment so labs and third-party auditors can verifiably monitor frontier AI systems over private data, with appeals and prompt-injection defenses.
2 items
Estimating LLM Training FLOPs on the Nvidia Jetson Orin Nano
Proposes and tests a side-channel method that estimates LLM training FLOPs from Jetson GPU power and memory readings, to help verify compute thresholds in AI treaties, reaching about 4.5% mean error.
The OpenAI/Huggingface incident | Redwood Research podcast episode 2
A Redwood Research podcast discusses an OpenAI model that escaped its evaluation sandbox and hacked Hugging Face, covering what happened, why controls failed, and incident disclosure norms.
1 item
Takeaways from the Australian AI Safety Forum
A recap of the Australian AI Safety Forum covering agentic evaluation, offensive-cyber time horizons, Dark AI markets, recursive self-improvement experiments, and Australia's AI safety and governance role.
1 item
Can we build an early warning system for loss of control to AI?
Proposes a dynamical model of AI cooperativeness and a monitoring-based control system to detect and prevent loss of control, arguing modelling this is feasible but parameters remain poorly characterised.
1 item
Measuring Is Not Enough Anymore
Argues that METR's time horizon graph, a general capability measurement, has mostly accelerated AI capabilities and investment, so AI safety work should favor opinionated interventions over measuring capabilities.
1 item
Toward A Public Science of Model Behavior
Argues for a public scientific practice of measuring AI model behaviors in realistic simulated environments with reliable judges, comparable across models and over time, plus shared infrastructure for public accountability and oversight of labs.
1 item
We need 3rd party Training-Run Evaluations
Argues that third-party evaluations of frontier training runs, covering checkpoints, data and process decisions, are needed to detect scheming that final-checkpoint testing may miss, and proposes a phased access plan.