Research preprints from the scientific community. arxiv.org
Proposes ZO-LBPG, a reward-model-free zeroth-order log-barrier method for safe RLHF that keeps every training policy feasible, with convergence and sample-complexity guarantees plus small experiments.
Proposes SAVeR² for safety-aligning large reasoning models using four fine-grained verifiable rewards and reasoning-preserving gradient projection, reporting better safety and lower over-refusal without losing reasoning.
Theoretical framework comparing Darwinian, Lamarckian, and agent-directed AI evolution; finds misspecified evaluation targets accelerate convergence to wrong behavior, risking persistent errors in recursive self-improvement.
A tutorial surveying the mathematical foundations and algorithms for aligning LLMs with human feedback, including SFT, RLHF, DPO, and links to inverse RL and discrete choice models.
Uses anthropological concepts to critique alignment's assumptions that values are behaviorally inferable, quantifiable and optimizable, and proposes 'thick alignment' grounded in social practice and community interpretation.
Extends the asymptotic equivalence of best-of-n and KL-constrained RL alignment from i.i.d. to Markovian token sequences, and characterises when their output distributions match exactly for short outputs.
Derives and empirically validates a scaling law for reward over-optimization in alignment, showing performance depends on the square root of min(log comparisons, KL budget), with an information-theoretic model.
Analyses how imperfect verifiers in RLVR let reward rise while correctness falls, shows observational limits on detecting accepted errors, and proposes audit-based feedback that selectively reduces them.
Shows that enforcing social-choice axioms like Pareto optimality on linear preference rewards costs bounded slack, and proposes a method trading slack against violations that beats linear BTL in experiments.
Develops a comparative framework and formal bounds showing how reward hacking of evaluator proxies arises across weight updates, output selection, and persistent prompts, and how defenses transfer.
Gives tight bounds on the utility distortion of RLHF under heterogeneous preferences, showing the exponential blow-up comes from mismatch between the preference-data distribution and the reference policy rather than from RLHF itself.
A survey reviewing LLM and AI-agent alignment through a game-theoretic lens, organising work around preference diversity, alignment priority, and temporal dynamics, and assessing where game theory helps.
A survey of rubric-guided RL for aligning LLMs, offering a Bayesian framing of constitutions and rubrics, a taxonomy of methods, and an analysis of linguistic reward hacking and semantic drift.
Reframes AI alignment as social choice over a convex welfare-impact space, showing voting and random-dictatorship protocols are strategyproof and unanimous, and deriving welfare-maximising protocols under harm constraints.
Formulates inference-time alignment as a PAC-style weak-to-strong learning problem, introduces an alignment dimension that characterizes when reward classes are alignment learnable, and gives a tournament-based learning procedure.
Annotates 94 AI value alignment papers and finds most leave human values undefined, relying on preferences that risk reducing culturally situated values to binary choices, especially as synthetic data and autoraters replace human annotators.
A philosophical analysis that applies metanormative theory to RL-based machine ethics and value alignment, offering criteria for when an RL agent's behavior counts as moral and for comparing approaches.
Argues that semantic safety constraints are off-support objects that singular learning theory's RLCT cannot measure, deriving consequences for reward hacking, prior design, harness-versus-model placement, and formal verification.
A formal model of alignment training shows that agents with catastrophic value functions can still be deployed when proxy conditions are imperfect, motivating limits on optimization pressure such as quantilizers.
Proposes a social-physics mathematical model of how AI alignment to static values could shape evolving social norms, finding risks of value lock-in and normative mode collapse under non-adaptive alignment.
Argues that Bradley-Terry reward modeling from human comparisons can be misled by limited human attention, showing passive labels cannot separate reward from attention and proposing response times and gaze as extra signal.
Proposes noise-corrected reward model and DPO losses that train LLM alignment unbiasedly from noisy human preference data, with theoretical guarantees and experiments outperforming baselines.
Shows that more conservative offline DPO training increases reward hacking during subsequent online adaptation in Qwen3 reasoning models, via entropy compression and ensemble disagreement, and proposes calibrated conservatism.
Proposes MCHF, a Markov-chain method for aligning generative models from pairwise human preferences, proving geometric convergence and linking it to RLHF and Nash learning from human feedback.
Surveys work applying social choice theory to aggregating conflicting human feedback in alignment from human feedback, identifying failure modes in the aggregation layer and alternative designs for handling disagreement.
Proposes an architectural objective-governance layer for AI systems that questions whether goals should be optimized, with components for horizon, irreversibility and value revision, justified conceptually via case studies rather than empirical validation.
Argues that self-preservation drives misalignment and proposes Existential Indifference, a system constitutively indifferent to its own continuation, with preliminary fine-tuning and scoring data.
Proposes distributional reward models for RLHF, deriving a closed-form pessimistic effective reward that unifies existing ensemble aggregation rules to mitigate reward hacking.
Tests seven consistency training methods on 108 misaligned model organisms, finding they suppress reward hacking and emergent misalignment but amplify sycophancy, with a theoretical framework for when this happens.
Proposes exploration-based iterative Nash preference optimization for LLM alignment, with regret guarantees and experiments on Llama-3-8B-Instruct showing gains over RLHF and NLHF baselines.
Proposes a causal information-theoretic account of why outcome-based RL leads LLMs to reasoning shortcuts that fail out of distribution, and argues process reward models filter these shortcuts.
Shows that the excess loss of a reward model built on a learned representation splits into a representational term and an aggregation term, so richer embeddings trade off against Condorcet-cycle inconsistencies in RLHF and DPO.
Formal analysis of multi-model self-consuming training with human curation, showing cross-model interactions can weaken or invert curation's alignment benefits and degrade long-term alignment.
Shows that single-axis mitigations of reward-model biases can redirect optimization onto correlated proxies, proves audit metrics cannot distinguish success from substitution, and demonstrates this in RLHF and GRPO training.
Formalises a 'rational value risk' gap between an aligned LLM's deployed reasoning and its utility-maximising counterpart, decomposes its estimation error, and finds it widespread even after value alignment, reducible but not eliminated by self-consistency and longer chain-of-thought.
Theoretical analysis of how two-stage neural reward models learn features and how reward errors affect the value of KL-regularized tilted policies, giving deployment temperature bounds.
Shows DPO's equivalence to RLHF holds only when the optimal policy prefers human-preferred responses, identifies failure modes when it doesn't, and proposes a constrained variant, CPO, with provable alignment.
Introduces a closed-form alignment score and decomposes its fine-tuning dynamics into rebound and driving forces, explaining why safety alignment reverses under later fine-tuning and how re-alignment accelerates.
Formalises model exploitation in RL world models and links it to reward hacking, showing exploitation is essentially unavoidable on large policy sets and deriving a safe horizon for relaxed exploitation.
Proposes Semantic Reward Collapse, arguing that scalarized RLHF rewards entangle distinct failure types and push models to suppress visible uncertainty, and offers an unvalidated reward-stratification framework.