AI research and discussion on LessWrong. www.lesswrong.com
Distinguishes minimal (jagged, fallible) from maximal (idealised) superintelligence, arguing they differ in timelines, risks and mitigations, and that prosaic safety measures matter for the near-term minimal case.
A dialogue arguing that a superintelligence could overcome security, physical and containment limits, using side-channel attacks, self-replicating systems and persuasion as examples.
Conceptual analysis of why corrigible AI agents should prioritize a principal's newer commands over older ones, examining temporal authority, empowerment, and decision policies.
Argues that AI oversight is currently necessary despite AI moral status, and that utility-maximising coherence can make an AI's defection incentives worse than incoherence, so many AIs may not want to scheme.
A speculative essay sketching non-extinction ASI futures, arguing alignment may be intractable and that multi-agent instrumental convergence and monitoring problems could push a principal toward confining its agents.
Argues that superintelligent text-only AI persuasion is limited by I/O constraints, proposes levels of superpersuasion, and assesses the plausibility of each, concluding arbitrary behavioural control is unlikely.
A conceptual taxonomy comparing introspective and extrospective recursive self-improvement, arguing that the internal route offers fewer monitoring channels and weaker verification, with implications for safety.
Proposes training agents to have preferences only between outcomes with equal numbers of copies (POSC), so they neither pay costs to self-replicate nor are strongly disposed to it, as a safeguard against AI self-replication.
The author argues formal verification mainly buys assurance between implementation and spec, not spec and world, so it should be treated as one defense-in-depth layer for AI security and boxing.
Carlsmith's essay examines how restricting AI rogue options (oversight, capability control, environmental hardening) and rewarding cooperation can prevent power-seeking, and the challenges of superhuman strategies in each.
Proposes training agents with incomplete preferences (POST) between different-length trajectories so they stay neutral about shutdown, proving this yields Neutrality+ and arguing it keeps agents shutdownable and useful.
A thought experiment using a lunar self-replicating robot colony to argue that self-sustaining, adaptive systems face compounding value drift toward self-preservation, and that control systems struggle to stay stable.
Argues that maintaining alignment during recursive self-improvement requires continual feedback control, with evaluations as sensors and interventions as actuators, rather than a single static value initialisation.
Proposes System 2 alignment techniques (review, deliberation, thought management) that developers likely adapt from capabilities work to align future LLM-based AGI agents, analyzing OpenAI's deliberative alignment as an example.
Barak argues that AI safety needs robust reasonable compliance with detailed specifications, defense in depth, and detection over prevention, and that it won't be solved by an AI scientist or by making AI love humanity.