AI research and discussion on LessWrong. www.lesswrong.com
Optimizes discrete token prefixes with GCG to imitate function vectors, filter heads and steering directions in Olmo, finding they shift behaviour more than direct injection and raising opaque-interface concerns.
Mechanistic analysis of Obfuscated Adversarial Training on Llama 3.2 checkpoints finds writer LoRAs aligned with harm probes, but stronger embedding attacks suppress the monitor-visible signal while harmful outputs persist.
Shows that activation steering, even with benign SAE or difference-in-means directions, can break LLM refusal in unpredictable ways, so legitimate users may unintentionally bypass safety behavior.
Replicates Stevinson et al.'s toy-model claims that adversarial vulnerability stems from superposition-induced feature interference, finding attacks align with optimal directions and transfer via shared geometry.
Proposes structural proxies, using present-day problems like adversarial fragility and unfaithful reporting to study future alignment challenges such as value generalisation and ELK, and assesses the approach's promise and limits.
A consumer-GPU replication of the role-confusion account of prompt injection finds steering and patching inconclusive, so whether the probed role axis is causal or a bystander stays unresolved.
Proposes that prompt injections exploit LLM role confusion, using role probes to show writing style overrides chat-template tags; introduces CoT Forgery, a jailbreak whose success tracks internal role perception.
Proposes calibrating activation steering vectors by residual-stream norm so one steering strength works across depths and models, shown to shift refusal rates on harmful-prompt benchmarks, and baking the steering into weights.
Uses cheap steering vectors with white-box access to remove refusals and CCP bias in Kimi K2.5, arguing that safety training is not resilient to weight-level attacks.
Replicates refusal-direction ablation on small LLMs and finds some models need a low-rank subspace, not one direction, to remove refusal, using weight orthogonalization.
Layer-wise analysis of Llama-3.1-70B finds jailbreak prompts show mid-layer harm signal that drops before output, unlike standard harmful prompts; the author notes structural confounds and calls for causal tests.
A monthly roundup of AI safety papers covering pretraining data filtering against harmful finetuning, reward hacking generalizing to misalignment, covert sandbagging against CoT monitors, safe-completions training, and probe-based defenses against cipher attacks.}
Shows via causal steering that LLMs encode harmfulness separately from refusal, finds jailbreaks often suppress refusal rather than harmfulness belief, and proposes a latent-representation guard.
Shows that steering vectors optimized on a single example can generalize across inputs to mediate alignment-faking behavior and refusal, achieving 96.9% attack success on HarmBench for Gemma-2-2B-it.
Uses gradients of the refusal metric with respect to early-layer activations and SAE latents to find upstream features causally driving refusal in Gemma-2-2b-it, validated by steering experiments.