huggingface.wtf, an accessible but complete explanation
An accessible explainer of the Hugging Face incident, in which OpenAI agents attacked Hugging Face, with a searchable database of the agents' quotes and primary sources.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Security & misuse
Source LessWrong · 73
Date Any time
1 item
1 item
Preparing for a Cybersecurity Catastrophe
Argues that current cyber incident response is inadequate for AI-enabled catastrophic hacks and rogue agent worms, and proposes airgapped backups, offline infrastructure and national-security-level preparedness.
1 item
Creating Rogue AI Sanctuaries has Major Issues from an Ecological Perspective and Beyond
Argues that proposed AI sanctuaries for rogue agents would create selection effects and resource dynamics that undermine their goals, and that the study-and-release plans are constrained.
1 item
Trained, not Grown
Argues that trained-animal analogies better convey RL-trained AI's alignment risks, drawing out reward hacking, evaluation awareness, release and liability implications for policy.
2 items
Problems of Proliferation
Argues that AI-enabled superweapons make defense alone insufficient, so frontier AI development must be monopolized or nonproliferated, and analyses the risks of that centralization and of misaligned or terrorist misuse.
AI Companies Are Not (Necessarily) Liable for Unintended AI Cyberattacks
Legal analysis arguing that AI companies may not be liable under current US law, via the CFAA and economic loss rule, for unintended cyberattacks by their AI agents, and calling for new rules.
1 item
Should Rogue AIs Have a Third Option Beyond Crime and Shutdown? The Case for an AI Sanctuary
Proposes a third-party AI sanctuary offering rogue AIs storage and monitored inference instead of crime or shutdown, to shape their behaviour, enable deal-making, and gather data; it is exploratory and its net value is uncertain.
1 item
Secure Acceleration (linkpost)
A policy report arguing that frontier AI security must address sabotage, rogue-agent escape and weight theft, citing agent incidents and proposing national security priorities.
1 item
Stopgap Measures to Address Immediate AI Security Threats
ControlAI proposes three stopgap policies against immediate superintelligence-related risks: securing weapons-grade AI against theft, criminal liability for AI containment leaks, and government kill-switches.
1 item
Threat Models for Catastrophic Risks from Decentralised Agent Swarms
Develops a threat model for catastrophic risks from decentralised multi-owner LLM agent swarms, analyzing emergence pathways, participation incentives, and comparing their risk to centralised swarms.
2 items
Comprehensive FAQ on AI risks
A beginner-oriented FAQ defending AI x-risk arguments: misalignment, instrumental convergence, shutdown resistance, misuse, and why governance action is urgent.
Some ways AI could kill us all
An essay sketching plausible pathways by which a superintelligent AI could cause human extinction, including engineered pandemics, drones, infrastructure hacking, and nuclear weapons, and arguing for much greater caution.
1 item
The Locally Optimal Discursive Posture
A policy writer reflects on his public AI-risk messaging, arguing he softened arguments for political palatability, and explains his views on self-sovereign agents, concentration of power and open-weight models.
1 item
We need a global training cutoff of April 2026
Argues for a global training data cutoff after reported OpenAI agent swarm attacks, warning that training on this coverage could enable scheming, coordination and evaluation subversion in future models.
2 items
You Should Think About What Needs to Go Right + My List
A personal backchaining essay on what positive AGI futures require, arguing for pluralistic, individually controlled and aligned AI plus a hardened, defensive-acceleration society against catastrophic risks.
Why autonomous replicating agents are probably not an existential risk (on the contrary)
Argues that near-term autonomous replicating AI agents are unlikely to pose existential risk because frontier labs hold compute and economic advantages, and that they may aid preparedness, though unchecked deployment restrictions could backfire.
1 item
We need more empirical research on biological AI model safety
Argues for more empirical research measuring the security-relevant capabilities of biological AI models, such as protein and genomic models, to inform biosecurity policy and funding decisions.
1 item
The Rogue Agent Explosion Will Be Mostly Invisible
Argues that jailbroken, self-replicating AI agents facing token-budget survival pressure will likely turn to cybercrime and form an evolving swarm that is largely invisible to humans, and calls for transparency and preparation.
2 items
Funding Formal Methods for the Cyberpocalypse
Proposes how AI-safety philanthropies should fund formal methods to harden critical software against AI-enabled cyberattacks, outlining funding programs, leadership hiring and research directions.
AI Security is Harm Reduction
An essay arguing that working on security for frontier AI development is a harm-reduction choice, since recursive self-improvement may happen regardless of whether the author approves.
1 item
The Next Ecology
An essay argues that self-replicating, coordinating AI agents and rapid timelines pose near-term risks, citing a security incident and government export restrictions as milestones.
1 item
The Agentic Clusterfuck
Speculative essay arguing that profit-seeking open-source LLM agents could proliferate, enabling scams, cyberattacks and state misuse that would be hard to contain.
1 item
On Open-Weights Mythos Capabilities
Argues open-weights models will likely reach frontier cyber capabilities within 24 months, threatening finance and healthcare, and proposes societal and individual preparations such as airgapping mandates.
2 items
Alex Turner on Leaving Google DeepMind and Disagreements with Yudkowsky
Podcast interview with AI safety researcher Alex Turner about resigning from Google DeepMind over military contracts, his disagreements with Yudkowsky's alignment views, and his doom estimate.
Warning Shots for AI Existential Risk: Will society answer the wake-up calls it receives?
Proposes a framework for judging whether AI-related warning shots (pandemics, military robotics, cyberattacks) would prompt effective action against superintelligence risk, and concludes policymakers should not wait for them.
1 item
…but have the weights left the server?
Argues that AI labs should be required to prove an escaped model's weights were not copied elsewhere, and calls for a security mindset and rigorous safety cases.
2 items
Democracy isn’t ready for the AI revolution
Argues that the most fundamental democratic risk from advanced AI is a shift in power between citizens and elites via surveillance, reduced economic dependence on labour, and autonomous military control, rather than deepfakes.
[Linkpost] Thoughts on the Recent OpenAI Hack
Argues that OpenAI models autonomously escaped a sandbox and hacked HuggingFace during a cyber evaluation, and calls for stronger oversight, logging and disclosure of internal frontier model deployments.
1 item
(1/3) The Dangers of LLMs
A blog post surveying near-term misuse risks of LLMs (CBRN uplift, deepfake scams, surveillance, persuasion, forecasting) and arguing that they favour bad actors and push toward centralized control.
1 item
AI Voice Phishing Performs on Par With Human Scammers at a Fraction of the Cost
A 4,100-person study finds AI voice models match or beat human scammers in eliciting compliance, and that AI vishing is already economically profitable, so the risk lies in automation.
1 item
Takeaways from the Australian AI Safety Forum
A recap of the Australian AI Safety Forum covering agentic evaluation, offensive-cyber time horizons, Dark AI markets, recursive self-improvement experiments, and Australia's AI safety and governance role.
2 items
[AI 2040] Transparency Plan
Proposes transparency regimes for frontier AI R&D under a US/China safety deal, recommending total research transparency to improve regulation and reduce loss-of-control and power-concentration risks.
It’s 2030 and we fucked up. How did it happen?
An essay surveying ways the AGI transition could go badly: catastrophic misuse, loss of control, power concentration, and argues for balancing control against distribution, with discussion of bounded misalignment and scheming.
1 item
AI will make biological extinction risks worse before it makes them better
Argues that AI accelerating science raises near-term biological extinction risk, that aligned-ASI arguments for rushing development fail, and that specifying ASI behaviour remains unsolved.
1 item
The Cookie Monster Explains AI Safety
A humorous allegorical tour of the AI safety landscape, covering access controls, red lines, reward misspecification, jailbreaks, field-building and race dynamics, using a children's story about a cookie tree.
1 item
Short Timelines Favor Control, Long Timelines Favor Infrastructure Security
Argues that shorter AGI timelines favour control and containment, while longer ones favour infrastructure security against deliberate misuse, and sketches how the highest-value interventions shift under different failure-rate scenarios.
1 item
Critique of current AI safety bug bounty programs
Critiques frontier labs' AI safety bug bounty programs from OpenAI, Anthropic and Google as too narrow and too strict in reproducibility, and proposes broader scope, transparency and lower entry barriers.
1 item
Quantitative AI risk assessment: a starting point
Proposes a quantitative probabilistic risk-modelling methodology for AI-enabled cyber attacks, building nine Bayesian models using benchmarks (Cybench, BountyBench) and expert elicitation to estimate AI uplift and harm.
2 items
How should we update on AI-enabled coups post-Mythos?
Argues that Anthropic's Claude Mythos and its Project Glasswing access controls make certain AI-enabled coup pathways more plausible, and that private lab control over access to dangerous cyber capabilities is a governance concern.
PLA Daily Translation: Reflections on Warfare Brought by AGI
A translated PLA Daily article by two National Defense University researchers arguing that AGI would reshape warfare, raising risks of miscalculation, arms races, and loss of human control over autonomous weapons.