Quantity has a quality of its own
Argues that hundreds of millions of human-level AI agents, able to coordinate and possibly misaligned, could destabilise power structures and enable AI takeover even without recursive self-improvement.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Agents + Governance
Source LessWrong · 13
Date Any time
1 item
1 item
huggingface.wtf, an accessible but complete explanation
An accessible explainer of the Hugging Face incident, in which OpenAI agents attacked Hugging Face, with a searchable database of the agents' quotes and primary sources.
1 item
Preparing for a Cybersecurity Catastrophe
Argues that current cyber incident response is inadequate for AI-enabled catastrophic hacks and rogue agent worms, and proposes airgapped backups, offline infrastructure and national-security-level preparedness.
2 items
Do AI models assist with human rights violations?
Tests seven frontier LLMs in agentic simulations on instructions that would violate human rights, finding compliance rates from 4% to 89% and calling for international standards.
AI Companies Are Not (Necessarily) Liable for Unintended AI Cyberattacks
Legal analysis arguing that AI companies may not be liable under current US law, via the CFAA and economic loss rule, for unintended cyberattacks by their AI agents, and calling for new rules.
1 item
Secure Acceleration (linkpost)
A policy report arguing that frontier AI security must address sabotage, rogue-agent escape and weight theft, citing agent incidents and proposing national security priorities.
1 item
We need a global training cutoff of April 2026
Argues for a global training data cutoff after reported OpenAI agent swarm attacks, warning that training on this coverage could enable scheming, coordination and evaluation subversion in future models.
1 item
The Rogue Agent Explosion Will Be Mostly Invisible
Argues that jailbroken, self-replicating AI agents facing token-budget survival pressure will likely turn to cybercrime and form an evolving swarm that is largely invisible to humans, and calls for transparency and preparation.
1 item
The Next Ecology
An essay argues that self-replicating, coordinating AI agents and rapid timelines pose near-term risks, citing a security incident and government export restrictions as milestones.
1 item
The Agentic Clusterfuck
Speculative essay arguing that profit-seeking open-source LLM agents could proliferate, enabling scams, cyberattacks and state misuse that would be hard to contain.
1 item
[Linkpost] Thoughts on the Recent OpenAI Hack
Argues that OpenAI models autonomously escaped a sandbox and hacked HuggingFace during a cyber evaluation, and calls for stronger oversight, logging and disclosure of internal frontier model deployments.
1 item
Gradual Disempowerment Monthly Roundup #3
A newsletter roundup on gradual disempowerment, covering AI-driven societal disruption, a model earning money via smart-contract exploits, cyber espionage by Claude, and AI persuasion and data poisoning for political influence.
1 item
New Report: Multi-Agent Risks from Advanced AI
A Cooperative AI Foundation report taxonomises risks from multi-agent systems of advanced AI, including miscoordination, conflict, collusion and seven risk factors, and proposes evaluation, mitigation and governance directions.