Distinguishes minimal (jagged, fallible) from maximal (idealised) superintelligence, arguing they differ in timelines, risks and mitigations, and that prosaic safety measures matter for the near-term minimal case.
Applies reliability theory to Google DeepMind's rogue-deployment defenses, showing how failure-domain structure sets rare-failure suppression and which component improvements matter most.
A dialogue arguing that a superintelligence could overcome security, physical and containment limits, using side-channel attacks, self-replicating systems and persuasion as examples.
Conceptual analysis of why corrigible AI agents should prioritize a principal's newer commands over older ones, examining temporal authority, empowerment, and decision policies.
Argues that AI oversight is currently necessary despite AI moral status, and that utility-maximising coherence can make an AI's defection incentives worse than incoherence, so many AIs may not want to scheme.
A speculative essay sketching non-extinction ASI futures, arguing alignment may be intractable and that multi-agent instrumental convergence and monitoring problems could push a principal toward confining its agents.
Proposes a variational framework for how much autonomy to delegate to AI systems and when human review is needed, with capacity theorems and simulations on oversight allocation.
Argues that AI safety should be treated as an epistemic property of an evolving learner rather than a snapshot behavioral property, introducing teachability as preserving future corrective leverage.
Uses control theory to prove that externally enforced AI safety strategies cannot sustain safety as capability grows, and that any viable alternative must be intrinsic and meet four structural requirements.
Argues that superintelligent text-only AI persuasion is limited by I/O constraints, proposes levels of superpersuasion, and assesses the plausibility of each, concluding arbitrary behavioural control is unlikely.
Shows classifier-based safety gates fail to reliably oversee self-improving controllers and LLM fine-tuning, while a Lipschitz ball verifier gives zero false accepts with analytical bounds.
A conceptual taxonomy comparing introspective and extrospective recursive self-improvement, arguing that the internal route offers fewer monitoring channels and weaker verification, with implications for safety.
Proposes GRACE, a neuro-symbolic architecture that separates deontic moral reasoning from agent decision-making and uses a Guard to enforce compliance, demonstrated on an LLM therapy assistant.
Reframes AI alignment as a decision-theoretic property and proposes MAP-AI, a Monte Carlo control-plane that selects policies by tail risk and misalignment probability without retraining models.
Proposes training agents to have preferences only between outcomes with equal numbers of copies (POSC), so they neither pay costs to self-replicate nor are strongly disposed to it, as a safeguard against AI self-replication.
The author argues formal verification mainly buys assurance between implementation and spec, not spec and world, so it should be treated as one defense-in-depth layer for AI security and boxing.
Carlsmith's essay examines how restricting AI rogue options (oversight, capability control, environmental hardening) and rewarding cooperation can prevent power-seeking, and the challenges of superhuman strategies in each.
Proposes training agents with incomplete preferences (POST) between different-length trajectories so they stay neutral about shutdown, proving this yields Neutrality+ and arguing it keeps agents shutdownable and useful.
Gives a formal, provable corrigibility framework for off-switch-style agents using lexicographically combined utility heads, with guarantees under learning error and an undecidability result for post-hack agents.
A thought experiment using a lunar self-replicating robot colony to argue that self-sustaining, adaptive systems face compounding value drift toward self-preservation, and that control systems struggle to stay stable.
Position paper arguing formal optimal control theory should underpin AI alignment, and introducing a layered Alignment Control Stack for controlling frontier and agentic models.
Argues that maintaining alignment during recursive self-improvement requires continual feedback control, with evaluations as sensors and interventions as actuators, rather than a single static value initialisation.
Proposes System 2 alignment techniques (review, deliberation, thought management) that developers likely adapt from capabilities work to align future LLM-based AGI agents, analyzing OpenAI's deliberative alignment as an example.
Barak argues that AI safety needs robust reasonable compliance with detailed specifications, defense in depth, and detection over prevention, and that it won't be solved by an AI scientist or by making AI love humanity.
Proposes a formal quantitative definition of alignment for autonomous agents in social decisions, derives conditions for near-optimal aligned policies, and a method to wrap black-box agents with verifiably safe actions.
Derives run-time probabilistic bounds on safety-violation risk under Bayesian hypothesis posteriors, proposing cautious-hypothesis selection to reject dangerous AI actions as guardrails.