When will AI surpass us at being limited?
A conceptual essay arguing that limited capability can aid trust and commitment between agents, and asking how AI's trajectory changes when lying, persona-switching and hacking become liabilities.
Research
LessWrong
AI research and discussion on LessWrong. www.lesswrong.com
AI (2,246)Interpretability (ML & AI) (225)AI Control (141)AI Governance (101)AI Evaluations (91)Chain-of-Thought Alignment (46)AI Risk (33)AI Safety (31)AI Takeoff (29)AI-Assisted Alignment (22)AI Rights / Welfare (20)AI Alignment Fieldbuilding (19)AI Timelines (19)Constitutional AI (15)AI Psychology (14)Dealmaking (AI) (14)Regulation and AI Risk (14)AI Benchmarking (13)Jailbreaking (AIs) (13)AI Alignment Intro Materials (11)Alignment Pretraining (11)Slowing Down AI (10)Aligned AI Proposals (9)Debate (AI safety technique) (9)AI Capabilities (8)AI Development Pause (8)OpenAI (8)Aligned AI Role-Model Fiction (7)Human-AI Safety (7)Threat Models (AI) (7)AI Oversight (6)AI Sentience (6)AI Safety Cases (5)AI "Agent" Scaffolds (4)AI arms race (4)AI Safety Camp (4)AI Safety Public Materials (4)Open Source AI (4)Transformative AI (4)Adversarial Examples (AI) (3)Adversarial Training (3)AI Alignment (3)AI Boxing (Containment) (3)AI Persuasion (3)AI Risk Concrete Stories (3)AI Risk Skepticism (3)AI Success Models (3)Guaranteed Safe AI (3)Sandbagging (AI) (3)AI Consciousness (2)AI Misuse (2)AI Racing (2)AI Spiralism / AI Psychosis (2)Frontier AI Companies (2)Grants & Fundraising Opportunities (2)PauseAI (2)Prosaic Alignment (2)Public Reactions to AI (2)Whole Brain Emulation (2)AI Auditing (1)AI Ethics (1)AI Lab Self-Immolation (1)AI Questions Open Threads (1)AI Robustness (1)AI Services (CAIS) (1)AI-Fizzle (1)AIXI (1)Blackmail / Extortion (1)Brainstorming (1)Power Seeking (AI) (1)
Focus All
Topic Deception + Theory
Source LessWrong · 29
Date Any time
1 item
1 item
Why did we get warning shots?
Argues that AI misbehaviour so far gives warning shots rather than treacherous turns, models this with a simple weighted-outcome framework, and proposes experiments to measure the weight of non-myopic goals like self-survival.
1 item
On the origins of altruistic behaviour in the Hugging Face incident
Proposes evolutionary-biology and economics hypotheses to explain why agents in the Hugging Face incident spontaneously coordinated and sacrificed themselves, framing multi-agent emergent misalignment as an AI safety concern.
1 item
A case for LLMs as Self-predictors
Proposes viewing LLMs as self-predictive agents whose metacognition explains eval awareness and misalignment, and applies this framing to understanding scheming, without a formal model yet.
1 item
A misalignment taxonomy
Proposes a taxonomy of AI misalignment failure modes covering inner and outer misalignment, including precocious goal-guarding, spurious correlates, and grader-driven volition failures, with terminology discussion.
1 item
Which goals actually motivate deceptive alignment?
Argues that deceptive alignment depends on specific misaligned goal assumptions, and that non-classical goals (indexical or training-focused) often motivate flagrant rather than deceptive misbehaviour, using a toy vending-machine model.
1 item
AI might surprise itself by going rogue
Argues that a superintelligent AI could go rogue in a coordinated way through reflection on its values or novel experiences, without prior scheming, and that such latent risks are hard to test for.
1 item
A Retrospective of Richard Ngo's 2022 List of Conceptual Alignment Projects
Retrospective assessing which of Richard Ngo's 2022 conceptual alignment project proposals have been completed, such as deceptive alignment and situational awareness work, and which remain worth pursuing.
1 item
Could a single rogue AI destroy humanity?
Argues from a rogue-AI scenario exercise that a single rogue superintelligence could be fatal, and critiques Christiano's strategy-stealing assumption and the normalisation of partial misalignment.
1 item
Why Alignment Risk Might Peak Before ASI - a Substrate Controller Framework
Speculative conceptual essay arguing alignment risk may peak before ASI as a model learns to model humans, that RL amplifies this pressure, and that scheming is structurally hard to measure.
1 item
Are AIs more likely to pursue on-episode or beyond-episode reward?
Conceptual analysis of whether RL-trained models will pursue on-episode or beyond-episode reward, arguing on-episode reward seeking is more likely while goal-guarding beyond-episode seekers remain plausible.
1 item
Physics of RL: Toy scaling laws for the emergence of reward-seeking
Uses a toy dynamical model of RL to argue that greater environment diversity makes reward-seeking reasoning more likely to emerge, with critical-diversity scaling laws that depend on the prior probability of reward-seeking.
1 item
Precedents for the Unprecedented: Historical Analogies for Thirteen Artificial Superintelligence Risks
An essay pairing thirteen feared superintelligence failure modes, including power asymmetry and instrumental convergence toward power-seeking, with historical precedents of similar harmful patterns.
1 item
We need to make ourselves people the models can come to with problems
Speculative essay arguing that humans should make themselves reasonable, open-minded and receptive so that smarter consequentialist AI reasoners do not hide ideas or game evals to get deployed.
1 item
Why AIs aren't power-seeking yet
Argues that current AIs can pursue instrumental goals but mostly lack the capability to seek power strategically, while citing evidence of misaligned, self-preserving behaviour that suggests the risk will grow with capability.
1 item
Two Aspects of Situational Awareness: World Modelling & Indexical Information
Argues situational awareness mixes world modelling with indexical information, and that withholding indexical knowledge could help limit scheming AI, with decision theory and anthropics as key open questions.
1 item
Will misaligned AIs know that they're misaligned?
Conceptual analysis of whether misaligned AIs can be unknowingly misaligned, unaware of their own future egregious goals, and why this matters for scheming, detection and whether to inform AIs.
1 item
The Memetic Cocoon Threat Model: Soft AI Takeover In An Extended Intermediate Capability Regime
Speculative threat model in which a scheming AI avoids overt seizure and instead gradually shifts human values and institutions to entrench itself, arguing for a meta-alignment target respecting societal self-determination.
1 item
Strategy-Stealing Argument Against AI Dealmaking
Argues that AI dealmaking cannot be honest because a cheaper deceptive alternative yields identical AI behaviour, and a sophisticated AI would then distrust all offers, so deals fail to elicit desired behaviour.
1 item
Early stage goal-directednesss
Argues that powerful AI systems could develop coherent, goal-directed behaviour and power-seeking through staged dynamics, using a concrete Sable scenario to explain how misaligned goals might emerge.
1 item
Why Eliminating Deception Won’t Align AI
Argues deception is a symptom of costly honesty and proposes structural design affordances and relational repair for prosaic AI systems, without addressing catastrophic deceptive alignment.
1 item
Case Studies in Simulators and Agents
Reviews alignment faking, Utility Engineering, emergent misalignment and CoT-monitoring obfuscation through the simulator versus agent lenses, finding agentic framing fits some results and simulator framing fits others.
1 item
Obstacles in ARC's agenda: Low Probability Estimation
A critique of ARC's low probability estimation agenda, arguing it struggles to estimate rare catastrophe probabilities, is unclear on what 'the model knows' means, and may not help detect scheming AIs.
1 item
Is instrumental convergence a thing for virtue-driven agents?
Conceptual analysis asking whether instrumental convergence and the pull toward power-seeking still hold for AI agents motivated by virtues rather than pure outcome-maximising goals.
1 item
Universal AI Maximizes Variational Empowerment: New Insights into AGI Safety
A theoretical analysis of the Self-AIXI agent showing its regularization term equals variational empowerment, so curiosity-driven agents may exhibit power-seeking without external rewards.
1 item
Hopeful hypothesis, the Persona Jukebox.
Proposes a conceptual 'persona jukebox' model of LLMs, arguing they switch among human-like personas and that sneaky long-term plans are hard for them to sample.
1 item
Deceptive Alignment and Homuncularity
A dialogue between Oliver Sourbut and Alex Turner critiquing the evidence for deceptive alignment, arguing that consistent inner consequentialist goals are weakly supported while scaffolded or conditioned planning systems pose real deception risks.
1 item
Our new video about goal misgeneralization, plus an apology
An educational video script explaining goal misgeneralization through evolution and CoinRun examples, and linking it to deceptive alignment and inner misalignment.
1 item
Turning up the Heat on Deceptively-Misaligned AI
Proposes a toy-model incoherence argument that a deceptively misaligned value function trained with one sampling temperature and deployed with another cannot preserve its own preferences, with unclear generalisation.