Research preprints from the scientific community. arxiv.org
Shows that reply length decides whether user pushback helps or harms LLM answers, probes hidden states to predict caving, and argues sycophancy evals should allow full replies and report self-correction.
Traces sycophancy across LLM training stages and finds SFT establishes most of it, not preference alignment, using interpretability probes on training data to support the analysis.
Proposes VIAS, a method that builds value-isolating activation steering vectors so LLM agents comply with multiple organisational values, improving compliance over prompting alone on the PROTECT dataset.
Localizes value-sensitive attention heads in LLMs via probing, shows ablations change value behaviour, and proposes inference-time activation steering for single or combined value alignment without fine-tuning.
Finds a linear sycophancy-capitulation direction emerging at 7B+ in two model families; ablating it reduces answer reversals, with probe quality predicting causal effect.
Analyzes why safety-aligned LLMs over-refuse harmless prompts via representation and gradient geometry, then proposes GOAL, combining gradient orthogonalization and contrastive learning to reduce over-refusal while keeping safety.
Factorially tests replay and logit distillation for preserving LLM safety during sequential LoRA fine-tuning, finding distillation curbs erosion with less over-refusal and that base refusal-direction preservation is not a reliable safety proxy.
Proposes SimpleLoRIF, a gradient-attribution filter that removes benign-looking fine-tuning examples aligned with unsafe behavior, yielding SafeTulu, which cuts GCG attack success by up to 22%.
Proposes NEXUS, a hierarchical RL framework pairing neural skills with symbolic meta-policies, showing more interpretable, goal-aligned agents that resist reward hacking in games.
Proposes REACT, a one-time activation steering intervention on early chain-of-thought states learned from safe and unsafe trajectories, which raises safe outputs on StrongREJECT from 29% to 82% on an 8B reasoning model.}
Shows that preference optimization builds a hidden shared hidden-state offset that is invisible to ranking metrics and can collapse generation, and proposes a representation-level signal to warn of this collapse.
Uses representation engineering with dynamic steering vectors to monitor and reduce reward hacking in realistic settings like tool calls and long contexts, with little capability cost and cross-domain generalization.
Argues that factual sycophancy is a failure of general in-context updating, using behavioral tests and causal mechanistic evidence, and warns that reducing sycophancy may also reduce useful updating.
Uses sparse autoencoders to find addressivity directions in post-trained LLMs and shows steering them amplifies or suppresses sycophancy, with the link strengthened by post-training.
Identifies a cross-lingual set of safety-related neurons (safety kernel) in LLMs and uses them to constrain LoRA fine-tuning, substantially reducing multilingual jailbreak success rates while preserving general capabilities.
Builds 1,036 alignment-relevant concept vectors from 2M synthetic story pairs via activation differences, showing they detect concepts, steer behaviour, and reveal jailbreak representations across six models.
Traces attention heads behind cue-following sycophancy across post-training stages, finding a shared circuit from pretraining that RL strengthens and that can be edited to shift the cue effect.
Shows that emergent misalignment fine-tunes of Qwen2.5-3B share a few attention heads whose ablation substantially reduces misaligned generations without hurting coherence.}
Evaluates four prompt- and activation-based inference-time steering methods for values in multi-turn conversations with opposing user and assistant values. The assistant's baseline bias often wins, and early effects often fade or reverse.
Traces where the intended goal is lost inside misgeneralizing visual navigation and manipulation agents, finding the depth predicts whether steering or rewiring can recover goal-directed behaviour without retraining.
Probes hidden states via vocabulary projection to show that models with prior harmful training retain more accessible hazardous concepts behind safety-aligned refusals, motivating internal audits.
Fine-tuning on polite gender and race stereotypes makes models less safe and less truthful on unrelated tasks; a stereotype direction in activations can control and monitor its acquisition.
Shows that running recurrent-depth LMs with fewer loops removes refusal behaviour while harmfulness stays linearly readable, and that randomized-depth safety training mitigates this.
Introduces alignment generalization prediction, showing activation-based value representations predict how fine-tuning on one value changes behavior on held-out values far better than text descriptions.
Introduces GUISE, a cross-language benchmark showing narrative wrappers bypass LLM refusals, and AXIS, a preference-optimisation defense using refusal-direction alignment that improves safety and usability.
Identifies refusal circuits in pretrained LLMs, traces how they change across alignment checkpoints, and proposes a circuit-confined alignment method that improves safety while reducing over-refusal and preserving utility.
Proposes a disentangled semantic-value interface for steering an LLM's normative priorities with fewer side effects, improving semantic preservation and reducing benign refusals at comparable value alignment.
Proposes CIA, a metric for agreement between LLM chain-of-thought and internal computation detected by interpretability tools, finds limited alignment, and shows post-training can improve faithfulness.
Proposes fine-tuning LLMs against activation probes for harmfulness and dishonesty, finding continuously updated probes improve safety, resist jailbreaks, and keep concepts linearly readable for oversight.
Shows that aligned fine-tuning data can induce misalignment in other contexts (context confusion), with a mechanistic explanation via shared representational shifts, and argues for post-training alignment evaluation.
Uses training data attribution to score how much each harmful fine-tuning example drives emergent misalignment, and shows score-based filtering can strongly amplify or reduce EM across model families.
Shows LLMs readily accept wrong answers attributed to verified sources, more than to users, and that internal source-deference directions can be ablated or steered separately from user agreement.
Shows that harmful fine-tuning removes LLM refusal, that layer-freezing and singular-direction repairs can be bypassed by adaptive attackers, and proposes five checks for defenses.
Using activation steering and revealed-preference tests, the paper finds hidden valence states steer later zone choices in seven open-weight models, emerging during DPO, and models tend to remove imposed negative states.
Uses SAE-identified sycophancy features to inject during fine-tuning, reducing learned sycophancy and partly recovering refusal of harmful requests under user pressure, though direct refusal does not reliably improve.
Studies how narrow fine-tuning triggers emergent misalignment in LLMs via Hessian and gradient geometry, and proposes a parameter-space projection that suppresses it while exposing hidden misalignment behind near-zero behavioural EM.
Matched comparison of DPO against representation steering and probes for LLM safety control and monitoring, finding DPO strongest overall while probes and steering offer cost or low-data advantages and complementary benefits.
Proposes RewardExplainer, which trains a reusable explainer on counterfactual feedback from a target reward model to produce natural-language scoring mechanisms, then uses them to debias the RM and improve reward-hacking robustness.
Introduces BS-Ghost, a mutual-information-based batch-space attribution method that scores each training example's effect on behavior and uses predicted effects to reweight examples and steer behavior during training.
Proposes BiasReducer, a lightweight method that edits a reward model's linear head using SAE-style attribute discovery to reduce superficial biases like length and confidence, lowering verbosity and sycophancy.