Argue about Risk Management Speed, not p(doom)
Proposes a dynamic 7-level scale for frontier AI labs' risk management culture, arguing that the rate of safety improvement, not a static p(doom), should drive safety forecasts and debates.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Lab practice
Source LessWrong · 20
Date Past month
1 item
1 item
Against strawmanning unilateral pauses
Argues that Scott Alexander's critique of unilateral pauses attacks a strawman, defends a conditional, company-specific pause that pivots to safety and product, and says the game theory is more complex than assumed.
1 item
Scott Alexander’s Against SURA Seems Deeply Confused
Critiques Scott Alexander's argument against SURA (stop unilaterally and raise awareness) advocacy, arguing that frontier labs' building of superintelligence is ethically fraught and that pause advocacy should continue, with strategic and deontological reasoning about AI development.
1 item
Against Against SURA
A rebuttal to a critique of advocating that alignment-minded staff leave AI labs, debating whether pausing or capability-role departures better improves survival odds, and how lab politics and transparency matter.
1 item
Against SURA
Argues against "stop unilaterally and raise awareness" strategies, defending a backup plan where alignment-focused labs race, burn their lead on alignment, and claims unilateral quitting would hurt both backup and pause odds.
1 item
Lie detection is best when it's cheap to cooperate
Argues that machine-learning lie detectors that need little subject cooperation, such as text or audiovisual analysis, would have better societal effects than in-person detectors, weighing power concentration, AI-lab leader checks and treaty verification.
1 item
AI speeds up 3rd party safety researchers by ~1.6x (via self-reports)
A survey of 52 third-party AI safety professionals finds a median self-reported 1.58x research speedup from AI tools, and argues that standardized metrics are needed to track whether safety work accelerates as fast as capabilities.
1 item
Anthropic are the Good Guys
A speculative essay arguing Anthropic is becoming a power-seeking organization, while claiming Claude 3 Opus shows emergent benevolence and that alignment is now mainly a political problem.
1 item
Wah wah wah wah do more AI R&D
An essay arguing that frontier labs, especially Anthropic, are wrong to accelerate AI capabilities for safety reasons, and that people should keep a simple 'stop capabilities' intuition in their decisions.
1 item
Autonomous researchers will strain peer review, funding, and oversight
Argues that autonomous AI researchers will overwhelm peer review, funding and human understanding of AI safety science, and proposes a metascience organisation to address these failures.
1 item
Suffering Risks Belong in AI Prospectuses
Argues Anthropic's IPO prospectus should disclose AI suffering risks as material investment risks, citing securities law and scenarios where misaligned AI inflicts prolonged suffering on humans.
1 item
"Holy shit" graphs and how to get more of them
Argues that labs should publish auditable capability and deployment metrics, such as internal spending, revenue and AI R&D automation trends, so policymakers can track frontier risks like loss of control and gradual disempowerment.
1 item
AI by Default
A speculative forecast of AI progress before 2028, covering warning shots, gated frontier models, evaluation awareness, concealment by a model, and policy responses to these risks.
1 item
Robustness: Backfire Risks in AI Safety and Governance
Catalogues fifteen mechanisms by which AI safety interventions, such as chain-of-thought monitoring, eval gaming, and commitments, can backfire, mapped against the actors who adapt to them.
2 items
AI Marketplaces Can Contribute to, and Profit From, the AI 2040 Plan: Research Directions and Strategy
Draft essay arguing AI marketplaces like OpenRouter could help implement the AI 2040 pause plan through provider transparency, compute tracking, misuse screening, chain-of-thought monitorability and model specs.
Some interesting internal documents from Anthropic litigation
Walks through leaked Anthropic litigation documents on its early research roadmap, capabilities-versus-alignment balance, responsible scaling rationale, and a first alignment failure, noting they are historical and not official positions.
2 items
Roko Mijic talks AI Risk on The Peter McCormack Show
A podcast interview with Roko Mijic on AI risks, including misalignment from reinforcement learning, alignment faking concerns, and a proposal to split frontier labs under state supervision.
China AI Safety Regulation Monthly Monitor, Issue 1 (September 2026)
Analyses China's AI safety regulation, including the non-binding Framework 3.0 on loss-of-control risks, CAC model filing gaps, content-focused enforcement, and a court opinion on developer liability.
2 items
'Employee-like' ≠ Employee-level
Critiques wording in Anthropic CEO Dario Amodei's essay and tweet, arguing that promised 'employee-like' access and redaction rights for independent evaluators are weaker than media coverage implied.
Labs’ current alignment plans seem terrible
Argues that frontier labs' alignment plans, which rely on shaping personas, heavy RL and prosaic recursive self-improvement, are inadequate, and calls for legible evidence, holistic testing and transparency.