Argue about Risk Management Speed, not p(doom)
Proposes a dynamic 7-level scale for frontier AI labs' risk management culture, arguing that the rate of safety improvement, not a static p(doom), should drive safety forecasts and debates.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Lab practice
Source LessWrong · 211
Date Any time
1 item
1 item
Against strawmanning unilateral pauses
Argues that Scott Alexander's critique of unilateral pauses attacks a strawman, defends a conditional, company-specific pause that pivots to safety and product, and says the game theory is more complex than assumed.
1 item
Scott Alexander’s Against SURA Seems Deeply Confused
Critiques Scott Alexander's argument against SURA (stop unilaterally and raise awareness) advocacy, arguing that frontier labs' building of superintelligence is ethically fraught and that pause advocacy should continue, with strategic and deontological reasoning about AI development.
1 item
Against Against SURA
A rebuttal to a critique of advocating that alignment-minded staff leave AI labs, debating whether pausing or capability-role departures better improves survival odds, and how lab politics and transparency matter.
1 item
Against SURA
Argues against "stop unilaterally and raise awareness" strategies, defending a backup plan where alignment-focused labs race, burn their lead on alignment, and claims unilateral quitting would hurt both backup and pause odds.
1 item
Lie detection is best when it's cheap to cooperate
Argues that machine-learning lie detectors that need little subject cooperation, such as text or audiovisual analysis, would have better societal effects than in-person detectors, weighing power concentration, AI-lab leader checks and treaty verification.
1 item
AI speeds up 3rd party safety researchers by ~1.6x (via self-reports)
A survey of 52 third-party AI safety professionals finds a median self-reported 1.58x research speedup from AI tools, and argues that standardized metrics are needed to track whether safety work accelerates as fast as capabilities.
1 item
Anthropic are the Good Guys
A speculative essay arguing Anthropic is becoming a power-seeking organization, while claiming Claude 3 Opus shows emergent benevolence and that alignment is now mainly a political problem.
1 item
Wah wah wah wah do more AI R&D
An essay arguing that frontier labs, especially Anthropic, are wrong to accelerate AI capabilities for safety reasons, and that people should keep a simple 'stop capabilities' intuition in their decisions.
1 item
Autonomous researchers will strain peer review, funding, and oversight
Argues that autonomous AI researchers will overwhelm peer review, funding and human understanding of AI safety science, and proposes a metascience organisation to address these failures.
1 item
Suffering Risks Belong in AI Prospectuses
Argues Anthropic's IPO prospectus should disclose AI suffering risks as material investment risks, citing securities law and scenarios where misaligned AI inflicts prolonged suffering on humans.
1 item
"Holy shit" graphs and how to get more of them
Argues that labs should publish auditable capability and deployment metrics, such as internal spending, revenue and AI R&D automation trends, so policymakers can track frontier risks like loss of control and gradual disempowerment.
1 item
AI by Default
A speculative forecast of AI progress before 2028, covering warning shots, gated frontier models, evaluation awareness, concealment by a model, and policy responses to these risks.
1 item
Robustness: Backfire Risks in AI Safety and Governance
Catalogues fifteen mechanisms by which AI safety interventions, such as chain-of-thought monitoring, eval gaming, and commitments, can backfire, mapped against the actors who adapt to them.
2 items
AI Marketplaces Can Contribute to, and Profit From, the AI 2040 Plan: Research Directions and Strategy
Draft essay arguing AI marketplaces like OpenRouter could help implement the AI 2040 pause plan through provider transparency, compute tracking, misuse screening, chain-of-thought monitorability and model specs.
Some interesting internal documents from Anthropic litigation
Walks through leaked Anthropic litigation documents on its early research roadmap, capabilities-versus-alignment balance, responsible scaling rationale, and a first alignment failure, noting they are historical and not official positions.
2 items
Roko Mijic talks AI Risk on The Peter McCormack Show
A podcast interview with Roko Mijic on AI risks, including misalignment from reinforcement learning, alignment faking concerns, and a proposal to split frontier labs under state supervision.
China AI Safety Regulation Monthly Monitor, Issue 1 (September 2026)
Analyses China's AI safety regulation, including the non-binding Framework 3.0 on loss-of-control risks, CAC model filing gaps, content-focused enforcement, and a court opinion on developer liability.
2 items
'Employee-like' ≠ Employee-level
Critiques wording in Anthropic CEO Dario Amodei's essay and tweet, arguing that promised 'employee-like' access and redaction rights for independent evaluators are weaker than media coverage implied.
Labs’ current alignment plans seem terrible
Argues that frontier labs' alignment plans, which rely on shaping personas, heavy RL and prosaic recursive self-improvement, are inadequate, and calls for legible evidence, holistic testing and transparency.
1 item
Towards embedded evaluations for scheming propensities
Proposes four claims a scheming safety case must substantiate and argues that third-party embedded evaluators need persistent, deep access to training, monitoring and chain-of-thought to verify them.
1 item
Downsides of a Mandatory Training-to-Internal Deployment Gap
Critiques a proposed rule delaying internal AI R&D use of new models, arguing it blocks real-world alignment feedback and invites exploitable safety-versus-capabilities distinctions, illustrated by a fictional shutdown-avoidance incident.
2 items
What is recursive self-improvement and what would it mean to ban it?
Surveys competing definitions of recursive self-improvement in frontier AI and argues that a ban is hard to operationalise, so labs should sharply limit AI authority over successor models.
Capabilities research expands the safety-usefulness Pareto frontier too
Argues that capabilities research can sometimes improve safety by reshaping the safety-usefulness Pareto frontier, while most capabilities research pushes developers toward less safety, with examples from RLVR, inference and burning the lead.
1 item
Principles for Embedded Evaluations
Apollo Research proposes principles for embedded external evaluations of frontier AI, centred on claim-based assessment of scheming risks with verdicts that reward openness and public reporting.
2 items
The world's best gradual disempowerment model organism: Frontier AI labs
An essay arguing frontier AI labs have been gradually disempowered by political, cultural and economic pressures toward accelerating capabilities, and that behavioral niceness is weak evidence of alignment.
“China needs more AI safety.” More of what, exactly?
A reflection on what 'more AI safety in China' should mean, covering China's algorithm registration regime, a CAC official's recognition of loss-of-control risk, Chinese safety research, and company engagement.
2 items
Even if others are less responsible you can still make things worse
Argues that AI labs staying in a development race can worsen outcomes even when others are less responsible, via artificial urgency, legitimising corner-cutting, undermining warnings, and diverting attention and funding.
The Alignment Community Is Unintentionally Building a Censor's Toolkit
Position paper arguing alignment methods are dual-use and already misused for censorship and manipulation, proposing transparency, verifiable alignment, model pluralism, and awareness as responses.
2 items
Evidence about risk should be transparent
Argues that third-party evaluators of loss-of-control risk should publish their underlying evidence so outside scientists and competitors can scrutinise it, rather than issuing unverifiable high-level judgments.
Plan R: AI Safety by ASICs
Proposes separating frontier AI training from deployment and enforcing the split with hardware-level ASIC inference, latency limits, and anti-dogfooding rules to reduce catastrophic AI risk.
1 item
"I am an AI Safety Researcher"
An essay on how safety researchers can avoid accelerating AI capabilities, critiquing interpretability, MIRI's past role, and frontier labs, and recommending information hygiene and career choices.
2 items
Stopgap Measures to Address Immediate AI Security Threats
ControlAI proposes three stopgap policies against immediate superintelligence-related risks: securing weapons-grade AI against theft, criminal liability for AI containment leaks, and government kill-switches.
The Cost of Utopias (a Dialog)
A dialogue defending Substrate Needs Convergence, the claim that AGI alignment is impossible because evolutionary pressures override engineered goals, and debating its political and philosophical implications for AI x-risk policy.
1 item
AI as orderly evacuation vs stampede
Argues that an orderly-evacuation analogy describes AI safety better than the arms-race framing, critiquing frontier labs' racing dynamics and the risk of stampede-like herd behaviour.
2 items
You Don’t Have to Trust the AI Labs (in order to take their call for regulation seriously)
An essay defending Anthropic CEO Dario Amodei's proposal for coordinated frontier AI slowdown and regulation, arguing for embedded independent evaluators and distributed trust rather than trust in labs.
How we might actually pace the frontier: A proposal for AI companies to do public pacing exercises.
Proposes that AI companies run public, independently evaluated pacing exercises such as multi-day training halts, to build verification capacity and prepare for larger frontier slowdowns.
1 item
There is a channel to 900M weekly users. What goes in it?
An opinion post arguing that Anthropic and OpenAI should use their chat interfaces to disclose recent AI incidents and the risks of frontier development to the public, with proposed message options.
2 items
What the Pro-Democracy Movement Knows About Quitting in Protest
Applies a pro-democracy movement framework on insider defections to frontier AI lab employees, arguing that quiet internal outreach and refusing specific work outperform public resignation or protest.
Categorical taboos are much better than threshold taboos: neuralese edition
Argues that neuralese should be governed by a categorical taboo rather than a numerical threshold, using nuclear and doping analogies, because chain-of-thought monitorability depends on keeping self-communication in natural language.