We urgently need to practice before we can pause
Argues a global AI pause needs incremental precedent, with third-party verification of companies' safety-motivated slowdowns building trust and assessor competence before a pause.
Research / Sources
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Strategy & forecasting
Date Any time
The best arguments arguments for "AI pause would significantly hurt the economy?"
Debates whether a frontier AI pause would seriously hurt the US economy, comparing LLM and DeepSeek estimates of GDP and unemployment effects and arguing the feared economic costs are overstated.
Is it better to directly or indirectly ban ASI?
Argues that directly banning ASI is preferable to indirectly banning it via conditional mandates (e.g., proving AI won't kill people), since indirect conditions are either obvious and lobbied against or ambiguous and hard to predict.
Against strawmanning unilateral pauses
Argues that Scott Alexander's critique of unilateral pauses attacks a strawman, defends a conditional, company-specific pause that pivots to safety and product, and says the game theory is more complex than assumed.
Scott Alexander’s Against SURA Seems Deeply Confused
Critiques Scott Alexander's argument against SURA (stop unilaterally and raise awareness) advocacy, arguing that frontier labs' building of superintelligence is ethically fraught and that pause advocacy should continue, with strategic and deontological reasoning about AI development.
Against Against SURA
A rebuttal to a critique of advocating that alignment-minded staff leave AI labs, debating whether pausing or capability-role departures better improves survival odds, and how lab politics and transparency matter.
Against SURA
Argues against "stop unilaterally and raise awareness" strategies, defending a backup plan where alignment-focused labs race, burn their lead on alignment, and claims unilateral quitting would hurt both backup and pause odds.
AI speeds up 3rd party safety researchers by ~1.6x (via self-reports)
A survey of 52 third-party AI safety professionals finds a median self-reported 1.58x research speedup from AI tools, and argues that standardized metrics are needed to track whether safety work accelerates as fast as capabilities.
Quantity has a quality of its own
Argues that hundreds of millions of human-level AI agents, able to coordinate and possibly misaligned, could destabilise power structures and enable AI takeover even without recursive self-improvement.
Anthropic are the Good Guys
A speculative essay arguing Anthropic is becoming a power-seeking organization, while claiming Claude 3 Opus shows emergent benevolence and that alignment is now mainly a political problem.
Power — a guide for the perplexed
Argues that power rests on self-reinforcing expectations, drawing on political science to sketch how AI systems could enable small groups to seize power and why democracies may become less stable.
Why Governments Might Race to Build ASI Even If It Could Kill Us All
Argues that realist geopolitics would push nations like the US and China to pursue aligned ASI for a decisive strategic edge, even if it poses existential risk, so bans are unlikely to hold.
Highlights from Garrison Lovely's "Obsolete"
A review of Garrison Lovely's book on AI's labor-displacement risks, arguing for pause advocacy, liability for uninsurable risks, and stronger governance against lobbying and unchecked deployment.
Wah wah wah wah do more AI R&D
An essay arguing that frontier labs, especially Anthropic, are wrong to accelerate AI capabilities for safety reasons, and that people should keep a simple 'stop capabilities' intuition in their decisions.
Greater good arguments in AI safety
Argues that prosaic AI safety work, such as chain-of-thought monitoring, reduces misalignment risk and should not be dropped in favour of a pause, and critiques the 'greater good' reasoning against it.
D̶o̶n̶’̶t̵⠀Can’t Look Up
Argues that AI development is hard to pause because of a visibility problem (unclear risks) and a structural problem (state incentives in anarchic international order), and proposes what coordination would need.
On Idealized Agency
Argues that agent foundations' idealized theories of agency, such as AIXI and UDT, are unlikely to yield strong safety guarantees once de-idealized, and that the research approach lacks empirical feedback and historical precedent.
Which AI economics and governance research most needs independent expert evaluation? (help The Unjournal prioritize it)
The Unjournal asks for input on prioritizing AI economics and governance research for public expert evaluation, covering growth forecasts, recursive self-improvement models, compute smuggling estimates, and AI liability work.
1000 Years of AI Safety
Models how much AI safety research a multi-year pause among top labs could produce, including verification bottlenecks and Plan A scaling tradeoffs, using a semi-endogenous growth framework.
What do the other 80 percent mean?
An Austrian author argues that public AI catastrophe estimates lack a breakdown of outcomes, cites prior work on decomposing AI risk, and proposes analog fallback infrastructure as preparation.
huggingface.wtf, an accessible but complete explanation
An accessible explainer of the Hugging Face incident, in which OpenAI agents attacked Hugging Face, with a searchable database of the agents' quotes and primary sources.
AI as a Normal Technology should be a vision
An essay arguing that AI should be governed as a potentially abnormal technology, critiquing the 'AI as Normal Technology' framing on loss-of-control risk, gradual disempowerment, and competitive pressure.
Quick Thoughts on Overhang Arguments
A rough critique of 'overhang' arguments in AI safety, which justify accelerating progress now to slow later risk, using analogies from other fields and arguing such reasoning is suspect and prone to motivated reasoning.
Preparing for a Cybersecurity Catastrophe
Argues that current cyber incident response is inadequate for AI-enabled catastrophic hacks and rogue agent worms, and proposes airgapped backups, offline infrastructure and national-security-level preparedness.
"Holy shit" graphs and how to get more of them
Argues that labs should publish auditable capability and deployment metrics, such as internal spending, revenue and AI R&D automation trends, so policymakers can track frontier risks like loss of control and gradual disempowerment.
AI by Default
A speculative forecast of AI progress before 2028, covering warning shots, gated frontier models, evaluation awareness, concealment by a model, and policy responses to these risks.
Robustness: Backfire Risks in AI Safety and Governance
Catalogues fifteen mechanisms by which AI safety interventions, such as chain-of-thought monitoring, eval gaming, and commitments, can backfire, mapped against the actors who adapt to them.
AI Marketplaces Can Contribute to, and Profit From, the AI 2040 Plan: Research Directions and Strategy
Draft essay arguing AI marketplaces like OpenRouter could help implement the AI 2040 pause plan through provider transparency, compute tracking, misuse screening, chain-of-thought monitorability and model specs.
Some interesting internal documents from Anthropic litigation
Walks through leaked Anthropic litigation documents on its early research roadmap, capabilities-versus-alignment balance, responsible scaling rationale, and a first alignment failure, noting they are historical and not official positions.
Agreement not to defect (alone) can mitigate loss of control risks
Argues that a widely endorsed public agreement against defecting in ASI development could deter races, since a cooperative future ASI might punish defectors, using game-theoretic reasoning about loss-of-control risk.
More people should write basic AI safety explainers
Argues that the AI safety community should write many more clear, basic explainers of the case for AI risk, since public awareness and clear communication are currently underinvested.
Labs’ current alignment plans seem terrible
Argues that frontier labs' alignment plans, which rely on shaping personas, heavy RL and prosaic recursive self-improvement, are inadequate, and calls for legible evidence, holistic testing and transparency.
ASI and Liberty: Can we do better than a benevolent god?
An essay arguing that AI-transition planning should preserve liberal-republican values such as non-domination and popular government, not just survival or aligned benevolent rule, and criticising both the democracy-focused and takeover-accepting camps.
How Would We Know? Reflections on trying to make AI go well amid deep uncertainty
A macrostrategy essay on AI safety under deep uncertainty, arguing for robust strategies, verification capacity and actionable epistemics, with estimates on alignment scaling, control failure, warning shots and governance regimes.
Downsides of a Mandatory Training-to-Internal Deployment Gap
Critiques a proposed rule delaying internal AI R&D use of new models, arguing it blocks real-world alignment feedback and invites exploitable safety-versus-capabilities distinctions, illustrated by a fictional shutdown-avoidance incident.
Creating Rogue AI Sanctuaries has Major Issues from an Ecological Perspective and Beyond
Argues that proposed AI sanctuaries for rogue agents would create selection effects and resource dynamics that undermine their goals, and that the study-and-release plans are constrained.
What is recursive self-improvement and what would it mean to ban it?
Surveys competing definitions of recursive self-improvement in frontier AI and argues that a ban is hard to operationalise, so labs should sharply limit AI authority over successor models.
Capabilities research expands the safety-usefulness Pareto frontier too
Argues that capabilities research can sometimes improve safety by reshaping the safety-usefulness Pareto frontier, while most capabilities research pushes developers toward less safety, with examples from RLVR, inference and burning the lead.
By default, Chinese AI falls very behind
Models China's incentives over the next 6-18 months for international AI risk coordination, arguing the US-China capability gap will likely widen without policy changes.
A simple argument about capabilities externalities versus opportunity costs
Argues that a typical AI safety project's capabilities externalities are usually smaller than the opportunity cost of not doing another generic safety project, so researchers should weigh opportunity costs more heavily.