Research / Sources
Research preprints from the scientific community. arxiv.org
Shows that loss-trained linear and some nonlinear reward models can violate Pareto optimality over principles, and that dataset design, not richer models alone, can recover it under idealized assumptions.
Uses an exactly computable RLVR setup and a covariance decomposition to show that cross-prompt parameter sharing can both suppress and amplify reward hacking, depending on verifier leak universality and step size.
Argues that AI alignment is ill-defined, unsolvable in many specifications, and insufficient because emergent risks in human-machine societies arise beyond individual misaligned systems.
Argues that frontier AI makes recursive self-improvement plausible, that no clear frameworks predict its outcomes, and offers a game-theoretic analysis of whether to start RSI when rivals might too.
Proposes a framework decomposing agent impact into intelligence, knowledge and control pathways, uses NK-landscape simulations of fast takeoff, and argues governance should restrict control and knowledge rather than intelligence.
Proposes ZO-LBPG, a reward-model-free zeroth-order log-barrier method for safe RLHF that keeps every training policy feasible, with convergence and sample-complexity guarantees plus small experiments.
Proposes SAVeR² for safety-aligning large reasoning models using four fine-grained verifiable rewards and reasoning-preserving gradient projection, reporting better safety and lower over-refusal without losing reasoning.
Links information-theoretic empowerment and capability-based human power via a goal-revelation bridge, showing empowerment-maximising or revelation-maximising assistants can fail, and proposing a satisfaction floor for monitoring retained human power.
Theoretical framework comparing Darwinian, Lamarckian, and agent-directed AI evolution; finds misspecified evaluation targets accelerate convergence to wrong behavior, risking persistent errors in recursive self-improvement.
Gives a formal definition of catastrophic performance from AI misalignment, using baseline and orthogonal reference values to separate catastrophic outcomes from merely suboptimal ones.
Introduces rewindable interactive proofs, letting a weak verifier rewind AI provers, giving a cryptography-free oversight protocol that does not assume a truthful prover, with an MNIST experiment.
Derives closed-form output-space gaps for gradient-based unlearning in linear regression, extended to nonlinear models, and proposes leverage diagnostics for auditing forget sets.
Proposes Probabilistic Agentic Programming, with a mediation kernel enforcing authority, approval and resource checks before LLM agent tool calls, proves conditional safety, and reports a small comparative study of observed task outcomes.
A tutorial surveying the mathematical foundations and algorithms for aligning LLMs with human feedback, including SFT, RLHF, DPO, and links to inverse RL and discrete choice models.
Develops a partition theory showing behavioural evaluations can only certify a floor on a model's discrimination, so detecting concealed capabilities or sandbagging via evaluation is fundamentally limited.
Proves that poisoning-based backdoors cannot be made cryptographically asymmetric: any verifier poisoning can install is reconstructible by a white-box defender, confirmed on instruction-tuned LLMs.
Formalises AI safety as a three-player game and proves a fundamental tradeoff: advice from an AI that helps a bounded defender while remaining public cannot avoid costs, and manipulating shared information to benefit only the defender is also examined.
Formal impossibility results show that no terminating gate or search can reliably accept only genuine self-improvements of agents in unbounded settings, while bounded settings with finite budgets can recover guarantees.
Argues that full value alignment of autonomous robots is theoretically and practically impossible, and proposes shifting from maximising alignment to defining conditions under which residual misalignment is acceptable, with normative requirements and a framework.
Uses anthropological concepts to critique alignment's assumptions that values are behaviorally inferable, quantifiable and optimizable, and proposes 'thick alignment' grounded in social practice and community interpretation.
Argues that human control over frontier AI is lost through organisational delegation and proposes checkable measures: named accountable persons, supervisors with stop authority, and published build-versus-check clocks.
Proposes an ecological theory of misaligned AI-agent populations showing a collaboration-driven critical population threshold for takeoff, and recommends ecological red teaming and population pacing.
Proves theoretical bounds showing that even a perfect agent cannot guarantee safe behaviour in partially observable environments with limited time to act, so safety proofs must consider the environment.
Proposes alignment scaling laws with burden exponents per risk category, proves regime results, and reports preregistered measurements on Pythia and Qwen models for adversarial robustness, truthfulness, sycophancy and backdoors.
Develops a non-operational failure-mode framework for how advanced AI could cause extinction or permanent disempowerment via misalignment, human misuse, organizational failure and competitive deployment, without requiring intent.
Position paper reframing human-agent alignment as ongoing interaction design, proposing five pillars (Specification, Process, Evaluation, Adaptation, Recalibration) for agents acting on users' behalf.
Develops a theory of defending LLMs against multi-turn mosaic attacks using a stateful watchman, with complexity bounds, and shows self-play red-teaming with LoRA adapters improves attacker and defender robustness.
Explains why subliminal trait transfer needs large datasets, casting steering-vector distillation as a noisy inverse problem where soft KL supervision recovers the trait far more efficiently than hard NLL labels.
Designs a new AI debate protocol for oversight with worst-case correctness and dominant-strategy honesty, and proves it is instance-optimal under black-box human judgment queries.
Proves that zero-knowledge verification of general oracle-aided AI computations is impossible in the random oracle model, including for debate, but shows signed oracle answers enable zero-knowledge verification as an alternative to scalable oversight.
Extends the asymptotic equivalence of best-of-n and KL-constrained RL alignment from i.i.d. to Markovian token sequences, and characterises when their output distributions match exactly for short outputs.
Models collective attacks by many LLM agents as a mean field game and derives an exact belief threshold below which no attack is the unique equilibrium, using a July 2026 incident as a worked example.
Derives and empirically validates a scaling law for reward over-optimization in alignment, showing performance depends on the square root of min(log comparisons, KL budget), with an information-theoretic model.
Derives a closed-form analysis of backdoor attacks on a quadratic neuron, showing feature learning lowers the trigger strength needed at small poison fractions, making feature learners more vulnerable.
Analyses how imperfect verifiers in RLVR let reward rise while correctness falls, shows observational limits on detecting accepted errors, and proposes audit-based feedback that selectively reduces them.
Formal ridgeless linear-regression and random-feature analysis of when a student trained on a teacher's pseudo-labels outperforms the teacher, governed by an active-bottleneck principle.
Shows that enforcing social-choice axioms like Pareto optimality on linear preference rewards costs bounded slack, and proposes a method trading slack against violations that beats linear BTL in experiments.
Introduces Evolutionary Safety, a taxonomy and framework for how safety properties persist or degrade through recursive self-improvement, with risk evaluation and governance principles for controlling and verifying updates.
Applies reliability theory to Google DeepMind's rogue-deployment defenses, showing how failure-domain structure sets rare-failure suppression and which component improvements matter most.
Develops a comparative framework and formal bounds showing how reward hacking of evaluator proxies arises across weight updates, output selection, and persistent prompts, and how defenses transfer.