Research preprints from the scientific community. arxiv.org
Shows that reply length decides whether user pushback helps or harms LLM answers, probes hidden states to predict caving, and argues sycophancy evals should allow full replies and report self-correction.
Traces sycophancy across LLM training stages and finds SFT establishes most of it, not preference alignment, using interpretability probes on training data to support the analysis.
Proposes VIAS, a method that builds value-isolating activation steering vectors so LLM agents comply with multiple organisational values, improving compliance over prompting alone on the PROTECT dataset.
Localizes value-sensitive attention heads in LLMs via probing, shows ablations change value behaviour, and proposes inference-time activation steering for single or combined value alignment without fine-tuning.
Finds a linear sycophancy-capitulation direction emerging at 7B+ in two model families; ablating it reduces answer reversals, with probe quality predicting causal effect.
Analyzes why safety-aligned LLMs over-refuse harmless prompts via representation and gradient geometry, then proposes GOAL, combining gradient orthogonalization and contrastive learning to reduce over-refusal while keeping safety.
Factorially tests replay and logit distillation for preserving LLM safety during sequential LoRA fine-tuning, finding distillation curbs erosion with less over-refusal and that base refusal-direction preservation is not a reliable safety proxy.
Proposes SimpleLoRIF, a gradient-attribution filter that removes benign-looking fine-tuning examples aligned with unsafe behavior, yielding SafeTulu, which cuts GCG attack success by up to 22%.
Proposes NEXUS, a hierarchical RL framework pairing neural skills with symbolic meta-policies, showing more interpretable, goal-aligned agents that resist reward hacking in games.
Proposes REACT, a one-time activation steering intervention on early chain-of-thought states learned from safe and unsafe trajectories, which raises safe outputs on StrongREJECT from 29% to 82% on an 8B reasoning model.}
Shows that preference optimization builds a hidden shared hidden-state offset that is invisible to ranking metrics and can collapse generation, and proposes a representation-level signal to warn of this collapse.
Uses representation engineering with dynamic steering vectors to monitor and reduce reward hacking in realistic settings like tool calls and long contexts, with little capability cost and cross-domain generalization.
Argues that factual sycophancy is a failure of general in-context updating, using behavioral tests and causal mechanistic evidence, and warns that reducing sycophancy may also reduce useful updating.
Uses sparse autoencoders to find addressivity directions in post-trained LLMs and shows steering them amplifies or suppresses sycophancy, with the link strengthened by post-training.
Identifies a cross-lingual set of safety-related neurons (safety kernel) in LLMs and uses them to constrain LoRA fine-tuning, substantially reducing multilingual jailbreak success rates while preserving general capabilities.
Builds 1,036 alignment-relevant concept vectors from 2M synthetic story pairs via activation differences, showing they detect concepts, steer behaviour, and reveal jailbreak representations across six models.
Traces attention heads behind cue-following sycophancy across post-training stages, finding a shared circuit from pretraining that RL strengthens and that can be edited to shift the cue effect.
Shows that emergent misalignment fine-tunes of Qwen2.5-3B share a few attention heads whose ablation substantially reduces misaligned generations without hurting coherence.}
Evaluates four prompt- and activation-based inference-time steering methods for values in multi-turn conversations with opposing user and assistant values. The assistant's baseline bias often wins, and early effects often fade or reverse.
Traces where the intended goal is lost inside misgeneralizing visual navigation and manipulation agents, finding the depth predicts whether steering or rewiring can recover goal-directed behaviour without retraining.
Probes hidden states via vocabulary projection to show that models with prior harmful training retain more accessible hazardous concepts behind safety-aligned refusals, motivating internal audits.