AI research and discussion on LessWrong. www.lesswrong.com
Optimizes discrete token prefixes with GCG to imitate function vectors, filter heads and steering directions in Olmo, finding they shift behaviour more than direct injection and raising opaque-interface concerns.
Mechanistic analysis of Obfuscated Adversarial Training on Llama 3.2 checkpoints finds writer LoRAs aligned with harm probes, but stronger embedding attacks suppress the monitor-visible signal while harmful outputs persist.
Shows that activation steering, even with benign SAE or difference-in-means directions, can break LLM refusal in unpredictable ways, so legitimate users may unintentionally bypass safety behavior.
Replicates Stevinson et al.'s toy-model claims that adversarial vulnerability stems from superposition-induced feature interference, finding attacks align with optimal directions and transfer via shared geometry.
Proposes structural proxies, using present-day problems like adversarial fragility and unfaithful reporting to study future alignment challenges such as value generalisation and ELK, and assesses the approach's promise and limits.
A consumer-GPU replication of the role-confusion account of prompt injection finds steering and patching inconclusive, so whether the probed role axis is causal or a bystander stays unresolved.
Proposes that prompt injections exploit LLM role confusion, using role probes to show writing style overrides chat-template tags; introduces CoT Forgery, a jailbreak whose success tracks internal role perception.
Proposes calibrating activation steering vectors by residual-stream norm so one steering strength works across depths and models, shown to shift refusal rates on harmful-prompt benchmarks, and baking the steering into weights.