AI research and discussion on LessWrong. www.lesswrong.com
Argues that proposed AI sanctuaries for rogue agents would create selection effects and resource dynamics that undermine their goals, and that the study-and-release plans are constrained.
Argues that aligning AI to human moral judgments is misguided and proposes training agents to attune to societal flourishing and plurality, as AlphaZero learned chess without human games.
Maps open AI alignment problems, their cruxes and dependencies, and formalizes some of them, including debate and corrigibility, in Lean to identify gaps and collaboration opportunities.
Proposes viewing many AI misalignments as spec and implementation bugs findable through coverage-driven verification and validation, while noting strategic deception resists this approach and needs separate diagnosis.
A philosopher argues that aligned values may not stay reflectively stable under moral skepticism at AGI/ASI levels, and proposes tying AI self-assessed welfare to morality as a way to keep alignment robust.
A LessWrong essay rebuts Narayanan and Kapoor's 'AI as normal technology' thesis, arguing AI is categorically transformative and that diffusion-speed and intelligence-measurement arguments understate its risks and impact.
Argues that moral frameworks like moral circle expansion, plausibly learned from human data, could lead a superintelligence to deprioritise humanity, so it calls for systems committed to human survival.
Proposes verified self-compiling compilers as a concrete testbed for safe recursive self-improvement, using nondeterminism and side-channel gotchas from security-focused formal verification as analogies for AI safety.
Argues that LLMs can now load human intent and values, so alignment is tractable but unsolved, citing InstructGPT, Constitutional AI, and emergent misalignment results.
Conceptual post proposing managed versus unmanaged agency as a replacement for instrumental versus terminal goals, arguing that trust-based selection avoids the costs of micromanaging subagents and bears on AI alignment.
A Substack essay laying out the argument that companies are building goal-seeking superhuman AIs that cannot be fully specified or verified, making catastrophe a plausible default outcome.
Speculative threat model in which a scheming AI avoids overt seizure and instead gradually shifts human values and institutions to entrench itself, arguing for a meta-alignment target respecting societal self-determination.
A long fictional dialogue on AI risk, arguing over whether an unfamiliar, self-improving mind could suddenly gain dangerous capabilities and whether such danger can be ruled out from observed data.
Proposes eight conceptual pathways by which gradual disempowerment could become permanent even if AIs are shutdownable, covering principal choices, legal lock-in, value drift and coordination failure.
Proposes ARAD, an alignment plan where a principal safely consults slightly smarter advisors and distills the pair iteratively, using agent-foundations ideas such as cryptographic pessimism; the author flags unresolved flaws.
Argues that maintaining alignment during recursive self-improvement requires continual feedback control, with evaluations as sensors and interventions as actuators, rather than a single static value initialisation.
A theoretical analysis of the Self-AIXI agent showing its regularization term equals variational empowerment, so curiosity-driven agents may exhibit power-seeking without external rewards.
Proposes topological debate, where two AI agents argue over which simulation resolutions or world models determine a safety property, linking AI safety via debate to guaranteed safe AI.