Research / Sources
AI research and discussion on LessWrong. www.lesswrong.com
A rough critique of 'overhang' arguments in AI safety, which justify accelerating progress now to slow later risk, using analogies from other fields and arguing such reasoning is suspect and prone to motivated reasoning.
Estimates how far zero-knowledge proofs of LLM inference are from cost-competitive with auditing for verifying AI compute in international agreements, measuring roughly 1,700-3,800x proving overhead and hash-function compatibility issues.
Literature review of proof of useful work, a cryptographic method for GPUs to prove compute use, as a way to verify international AI pause treaties, finding open gaps in guarantees and hardware assumptions.
Proposes making AI hardware resemble bounded humans, such as fused-weight ASICs, so existing institutions stay workable, arguing this is more robust than alignment-only approaches.
Proposes separating frontier AI training from deployment and enforcing the split with hardware-level ASIC inference, latency limits, and anti-dogfooding rules to reduce catastrophic AI risk.
Argues against U.S. export controls on AI chips as a misalignment-reducing policy, contending they may speed U.S. recursive self-improvement and hinder a bilateral treaty with China.
Argues that per-run FLOP caps for frontier training can be covertly bypassed by smuggling weights into new runs or aggregating parallel runs, and suggests a global training budget instead.
Formalises AI doom versus alignment deliverance as a competing-risks survival model driven by compute allocation and pacing, examining how response-function shapes change whether speed or split matters.
MIRI shares reading lists and research advice for newcomers to technical AI governance, drawn from its 2026 fellowship, to help researchers enter the field.
Proposes a rapid crisis playbook for pacing frontier AI before recursive self-improvement, covering an interim US-China deal, a voluntary company charter, verification with existing tools, and prep work.
Proposes designing an enforceable AI pause around a breakout-time threshold, arguing it should be set where rogue actors would need years to reach dangerous capabilities, with adaptive adjustment and supply-chain controls.
Proposes a shared, public-cluster observatory of GPU telemetry and other physical signals to verify AI training runs, as a first measurement layer for frontier AI governance.
Analyses US federal statutes and executive powers that let the president unilaterally influence or control frontier AI companies, and the limits imposed by Congress and courts.
Proposes concrete domestic US policies to pace frontier AI development, including compute allocation minimums for safety research, caps on AI R&D use, and safety-case-based existential risk thresholds enforced by third-party assessors.
Proposes a network-based verification design, traffic shaping plus randomized routing, that estimates it would make covert frontier training runs economically infeasible and so support treaty compliance.
Lays out five counterintuitive takeaways from the Plan A AI slowdown proposal, covering safety effort versus slowdown time, transparency, compute arms control, deal-breakdown risk, and the security-transparency tradeoff.
Argues that an international AI slowdown can be verified with human inspectors given whole-lab access to frontier labs, and that the technical verification barrier is not a reason to delay it.
Analyses technical mechanisms for verifying that datacenter compute is used only for inference, such as interconnect limits, memory wipes, replay, and zero-knowledge proofs, to slow unmonitored frontier training.
Proposes a taxonomy for verifying and enforcing international AI pauses via compute monitoring, estimating that pauses are enforceable for training runs above roughly 10k H100-equivalents absent concealment.
A commentary on the AI-2040 'Plan A' scenario, a proposed international treaty to slow AI progress, assessing its governance, enforcement, and alignment assumptions, including reliance on AI lie detection.
Proposes and tests a side-channel method that estimates LLM training FLOPs from Jetson GPU power and memory readings, to help verify compute thresholds in AI treaties, reaching about 4.5% mean error.
An essay arguing that AI power and control should be decentralized rather than concentrated in a few firms, governments, or the AI itself, and that democratic checks must be built into how AI is deployed.
Summarises AI 2040: Plan A, an international AI-race slowdown treaty with compute verification and transparency that still funds alignment research, and lists its failure modes.
Brainstorms governance options for advanced AI development, including chip-level monitoring, joint projects, deterrence, capability red lines, and liability regimes, and how they could combine into a stronger strategy than Plan A.
A research agenda from the AI Futures Project listing open questions on Plan A for navigating the intelligence explosion, covering alternative scenarios, covert projects, US domestic governance, verification, economics and geopolitics.
Critiques the AI Futures Project's Plan A pause-and-compute strategy, arguing that accumulated compute and software overhang (dry tinder) could make a deal breakdown far more dangerous, and that mutually assured compute destruction is fragile.
A quantitative model of how destroying US and Chinese AI compute would delay superintelligence development, finding strikes cause only 1-5 year delays that nationalisation can offset, undermining kinetic deterrence.
Models how much a 10x cut in AI R&D compute slows takeoff, deriving a slowdown factor of about 6x and estimating it with the AI Futures Model.
Proposes how to slow AI progress if needed, arguing for layered compute caps and taxes on R&D and deployment, timed to critical capability windows while managing overhang risk.
Proposes a low-trust reference architecture for verifying AI compute, using network taps, air-gapped or ZKP-based evaluation, and workload evidence capture to check treaty compliance.
Examines whether Trusted Execution Environments can give verifiable, privacy-preserving enforcement and monitoring of AI deployments for governance, finding hardware auditability to be the main obstacle.
Argues that compute verification is key to AI coordination, that die-level verification is too slow for short timelines, and that board- and firmware-level verification for Rubin and Feynman chips is more tractable and must start now.
Argues that FLOP/s and bandwidth proxies no longer track AI compute well and proposes architecture-specific compute scaling models, to improve forecasts and compute governance thresholds.
An essay arguing that AGI-making know-how and cheap compute are spreading, so powerful AI may soon be reproducible by many actors, and that this proliferation needs governance attention.
Argues that an international treaty to stop frontier AI via dismantling the compute supply chain is easier to enforce than nuanced regulation, analysing monitoring, standards, and violation challenges.
Proposes a tweak to MIRI's international AI pause agreement that closes a distributed-training loophole by also covering clusters with over 1,280 GB of accelerator memory, to make secret training runs detectable.
A blog post arguing that AI risk could be addressed by internationally dismantling the advanced chip supply chain, covering fabs, lithography, and datacenters, and weighing its risks against surveillance-based approaches.
Carlsmith analyses when and how capability restraint for AI safety is feasible and desirable, covering game-theoretic framing, international regimes, and trade-offs such as power concentration and enabling safety research.
Argues that US frontier AI companies would be unable to relocate abroad because the executive branch could block the move using export controls and IEEPA, removing relocation as a bargaining chip against regulation.
A workshop report on building hardware-based verification mechanisms, a datacenter 'lie detector', to check AI compute use for international agreements and reduce arms-race risks.