We urgently need to practice before we can pause
Argues a global AI pause needs incremental precedent, with third-party verification of companies' safety-motivated slowdowns building trust and assessor competence before a pause.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic International
Source LessWrong · 164
Date Any time
1 item
1 item
Against strawmanning unilateral pauses
Argues that Scott Alexander's critique of unilateral pauses attacks a strawman, defends a conditional, company-specific pause that pivots to safety and product, and says the game theory is more complex than assumed.
1 item
Scott Alexander’s Against SURA Seems Deeply Confused
Critiques Scott Alexander's argument against SURA (stop unilaterally and raise awareness) advocacy, arguing that frontier labs' building of superintelligence is ethically fraught and that pause advocacy should continue, with strategic and deontological reasoning about AI development.
1 item
Against Against SURA
A rebuttal to a critique of advocating that alignment-minded staff leave AI labs, debating whether pausing or capability-role departures better improves survival odds, and how lab politics and transparency matter.
1 item
Against SURA
Argues against "stop unilaterally and raise awareness" strategies, defending a backup plan where alignment-focused labs race, burn their lead on alignment, and claims unilateral quitting would hurt both backup and pause odds.
1 item
Lie detection is best when it's cheap to cooperate
Argues that machine-learning lie detectors that need little subject cooperation, such as text or audiovisual analysis, would have better societal effects than in-person detectors, weighing power concentration, AI-lab leader checks and treaty verification.
1 item
Anthropic are the Good Guys
A speculative essay arguing Anthropic is becoming a power-seeking organization, while claiming Claude 3 Opus shows emergent benevolence and that alignment is now mainly a political problem.
1 item
Why Governments Might Race to Build ASI Even If It Could Kill Us All
Argues that realist geopolitics would push nations like the US and China to pursue aligned ASI for a decisive strategic edge, even if it poses existential risk, so bans are unlikely to hold.
1 item
D̶o̶n̶’̶t̵⠀Can’t Look Up
Argues that AI development is hard to pause because of a visibility problem (unclear risks) and a structural problem (state incentives in anarchic international order), and proposes what coordination would need.
1 item
How far is ZKP from compute verification in high-stakes environments?
Estimates how far zero-knowledge proofs of LLM inference are from cost-competitive with auditing for verifying AI compute in international agreements, measuring roughly 1,700-3,800x proving overhead and hash-function compatibility issues.
1 item
AI Marketplaces Can Contribute to, and Profit From, the AI 2040 Plan: Research Directions and Strategy
Draft essay arguing AI marketplaces like OpenRouter could help implement the AI 2040 pause plan through provider transparency, compute tracking, misuse screening, chain-of-thought monitorability and model specs.
1 item
Agreement not to defect (alone) can mitigate loss of control risks
Argues that a widely endorsed public agreement against defecting in ASI development could deter races, since a cooperative future ASI might punish defectors, using game-theoretic reasoning about loss-of-control risk.
1 item
Proof of useful work for verifying AI treaties
Literature review of proof of useful work, a cryptographic method for GPUs to prove compute use, as a way to verify international AI pause treaties, finding open gaps in guarantees and hardware assumptions.
1 item
Book Review: Obsolete
A review of Garrison Lovely's book Obsolete, which covers AI catastrophic risk, power concentration, antitrust and policy prescriptions, and the reviewer's disagreements with parts of it.
2 items
A potential post-AGI world government
A speculative essay arguing that a post-AGI world needs multiple checked military powers and AI-aided democratic feedback to avoid both x-risk and dangerous power concentration.
By default, Chinese AI falls very behind
Models China's incentives over the next 6-18 months for international AI risk coordination, arguing the US-China capability gap will likely widen without policy changes.
4 items
Problems of Proliferation
Argues that AI-enabled superweapons make defense alone insufficient, so frontier AI development must be monopolized or nonproliferated, and analyses the risks of that centralization and of misaligned or terrorist misuse.
Do AI models assist with human rights violations?
Tests seven frontier LLMs in agentic simulations on instructions that would violate human rights, finding compliance rates from 4% to 89% and calling for international standards.
RAND's extinction report estimates one side of an inequality
Argues the RAND extinction report ignores coordination lag relative to risk maturation, and that pre-emptive coordination before capability jumps is needed for AI risk policy.
“China needs more AI safety.” More of what, exactly?
A reflection on what 'more AI safety in China' should mean, covering China's algorithm registration regime, a CAC official's recognition of loss-of-control risk, Chinese safety research, and company engagement.
1 item
Plan R: AI Safety by ASICs
Proposes separating frontier AI training from deployment and enforcing the split with hardware-level ASIC inference, latency limits, and anti-dogfooding rules to reduce catastrophic AI risk.
1 item
Against Export Controls (and China Threat Models)
Argues against U.S. export controls on AI chips as a misalignment-reducing policy, contending they may speed U.S. recursive self-improvement and hinder a bilateral treaty with China.
1 item
The AI race is already multipolar
Proposes an eight-outcome model of the superintelligence race, arguing that catastrophic misalignment and gradual disempowerment are key risks and that actors with conflicting goals can still cooperate to reduce them.
1 item
Pacing the Frontier: A Framework & Research Agenda
Proposes pacing AI development, meaning interventions that moderate its pace, as a dedicated research field, covering incentives, regulatory and technical tools, and the full lifecycle of interventions.
1 item
The Most Important City in AI Safety May Be Singapore
Argues Singapore is a strategic hub for AI safety field-building, international US-China coordination, and verification and evals work, and lists local organisations and next steps.
1 item
You Don’t Have to Trust the AI Labs (in order to take their call for regulation seriously)
An essay defending Anthropic CEO Dario Amodei's proposal for coordinated frontier AI slowdown and regulation, arguing for embedded independent evaluators and distributed trust rather than trust in labs.
2 items
My recommended resources for AI safety, alignment, and existential risks
A personal curated list of AI safety resources, covering foundational papers, videos, podcasts, organisations, policy statements and expert surveys on existential risk.
How a cold email got the Finnish government to respond on superintelligence regulation
A volunteer describes cold-emailing a Finnish MP, who prompted a parliamentary question and a government response on international regulation to prevent superintelligence development.
1 item
The Scramble: getting in position to pace the frontier
Proposes a rapid crisis playbook for pacing frontier AI before recursive self-improvement, covering an interim US-China deal, a voluntary company charter, verification with existing tools, and prep work.
1 item
AI breakout time: a mechanism for an enforceable AI pause
Proposes designing an enforceable AI pause around a breakout-time threshold, arguing it should be set where rogue actors would need years to reach dangerous capabilities, with adaptive adjustment and supply-chain controls.
2 items
Open Compute Verification Observatory as a First Metrological Layer for Technical AI Governance
Proposes a shared, public-cluster observatory of GPU telemetry and other physical signals to verify AI training runs, as a first measurement layer for frontier AI governance.
Let's talk about the AI coordination problem
Argues that coordination among AI developers to avoid building dangerous AI deserves serious, concrete discussion like a real geopolitical negotiation, and that leaders should explain why they consider it infeasible.
1 item
AI Safety in Japan has deeper problems than capital allocation
An opinion piece arguing that Japan's AI safety efforts lack a clear field strategy compared with Singapore and the Philippines, which have pivoted to AI verification and governance work, and urging Japanese institutes to set priorities.
1 item
Misaligned Incentives in Pause Scenarios
A LessWrong dialogue weighing whether a pause or slowdown of frontier AI development would be good, covering regulation incentives, political capacity, and timelines for a safe transition.
1 item
Will There Be an AI Hegemon? A Mental Model for AI Power Concentration
Proposes a mental model of AI power concentration, arguing a single actor could become a hegemon if it gains power faster than rivals can coalesce, and suggests tracking revenue growth and compute shares.
1 item
Did the alignment community underestimate its power?
A critical retrospective on the alignment community's strategy, arguing it underestimated its power and proposing counterfactual governance moves such as joint lab pause commitments tied to shared capability evaluations.
1 item
On Democratizing ASI to Preserve Civil Liberties
An essay arguing that ASI threatens liberal democracy and power balance, proposing gradual distribution of AI capabilities as a contingency, while maintaining that an international pause is preferable.
2 items
Traffic Shaping for Workload Classification
Proposes a network-based verification design, traffic shaping plus randomized routing, that estimates it would make covert frontier training runs economically infeasible and so support treaty compliance.
Five counterintuitive insights from Plan A
Lays out five counterintuitive takeaways from the Plan A AI slowdown proposal, covering safety effort versus slowdown time, transparency, compute arms control, deal-breakdown risk, and the security-transparency tradeoff.
1 item
Alex Turner on Leaving Google DeepMind and Disagreements with Yudkowsky
Podcast interview with AI safety researcher Alex Turner about resigning from Google DeepMind over military contracts, his disagreements with Yudkowsky's alignment views, and his doom estimate.