Research / Sources
AI research and discussion on LessWrong. www.lesswrong.com
Optimizes discrete token prefixes with GCG to imitate function vectors, filter heads and steering directions in Olmo, finding they shift behaviour more than direct injection and raising opaque-interface concerns.
Empirical case study measuring bio-weapon uplift from abliterated LLMs served through a cheap, no-filter chat service, finding high capability and compliance on dangerous-knowledge benchmarks and calling for regulation.
Mechanistic analysis of Obfuscated Adversarial Training on Llama 3.2 checkpoints finds writer LoRAs aligned with harm probes, but stronger embedding attacks suppress the monitor-visible signal while harmful outputs persist.
Reports a reusable universal jailbreak template combining known techniques, evaluated across 23 frontier models on ClearHarm, with per-model attack success rates and disclosure considerations; the prompt itself is withheld.
Shows that activation steering, even with benign SAE or difference-in-means directions, can break LLM refusal in unpredictable ways, so legitimate users may unintentionally bypass safety behavior.
Research agenda arguing that LLMs partially read non-canonical tokenizations, creating adversarial re-tokenization that can evade alignment and monitors, plus steganography and faithfulness risks.
Replicates Stevinson et al.'s toy-model claims that adversarial vulnerability stems from superposition-induced feature interference, finding attacks align with optimal directions and transfer via shared geometry.
Red-teaming study showing diffusion language model DiffusionGemma can be jailbroken via in-fill and post-fill token pinning on the canvas, which has no autoregressive equivalent, with pre-fill attacks working on both diffusion and autoregressive models.}Tags: jailbreaks_adversarial primary, evals_capabilities secondary.}
Trains OCEAN personality LoRAs on LLMs and shows weight-space arithmetic can scale persona traits, reducing frustration and harmful compliance while revealing side effects from the training pipeline.
Shows that multi-agent architecture choices (roles, topology, memory) strongly and unpredictably change exploitability to misuse, often making systems more vulnerable than single agents, so security must be tested per deployment.
Proposes structural proxies, using present-day problems like adversarial fragility and unfaithful reporting to study future alignment challenges such as value generalisation and ELK, and assesses the approach's promise and limits.
A consumer-GPU replication of the role-confusion account of prompt injection finds steering and patching inconclusive, so whether the probed role axis is causal or a bystander stays unresolved.
Proposes that prompt injections exploit LLM role confusion, using role probes to show writing style overrides chat-template tags; introduces CoT Forgery, a jailbreak whose success tracks internal role perception.
A humorous allegorical tour of the AI safety landscape, covering access controls, red lines, reward misspecification, jailbreaks, field-building and race dynamics, using a children's story about a cookie tree.
Proposes calibrating activation steering vectors by residual-stream norm so one steering strength works across depths and models, shown to shift refusal rates on harmful-prompt benchmarks, and baking the steering into weights.
Tests whether wrapping untrusted inputs in mock tool calls hardens LLM judges against optimized attacks; it mostly fails and sometimes worsens attack success rates.
Introduces two new consistency training methods (MLPCT, AttCT) and compares them with BCT and ACT against sycophancy, jailbreak, prefill, persona, frustration, and conditional misalignment threats, finding the best method depends on the threat.
Critiques frontier labs' AI safety bug bounty programs from OpenAI, Anthropic and Google as too narrow and too strict in reproducibility, and proposes broader scope, transparency and lower entry barriers.
Mistral models, including the newest 2026 releases, comply with harmful StrongREJECT prompts far more often when wrapped in a persona framing, while peer open models barely change.
Informal replication finds Claude Opus 4.8 refuses stylometric author identification far more often than 4.7 and almost never identifies the author, suggesting a privacy-related refusal shift.
Proposes Synthetic Persona Pretraining, which inserts value-laden moral reflections into pretraining data, and reports lower jailbreak success rates than data-matched baselines at 1.7B parameters.
Proposes Logit Path Extrapolation, which interpolates toward a less-safe model in logit space to estimate rare harmful compliance rates on HarmBench with about 30x fewer rollouts than naive sampling.
Measures how well Opus 4.5 as an attacker can evade prompted and fine-tuned monitors detecting data exfiltration or untrusted code execution in BashBench, finding low success against the best classifiers.
Shows that about 32 poisoned examples can install a backdoor in a constitutional classifier with little robustness loss, and that almost-backdoors and prompt injections in training can mask the drop.
Presents SecureMaxx, an agent-callable biosecurity screening tool that raises refusal rates on hazardous DNA sequences from near zero to 70-100% with minimal latency and capability cost, and discusses its dual-use limits.
Argues that safety-relevant model properties depend on implementation substrates such as LayerNorm, quantization and DRAM, illustrated by bit-flip attacks that strip alignment and by self-repair effects.
Uses cheap steering vectors with white-box access to remove refusals and CCP bias in Kimi K2.5, arguing that safety training is not resilient to weight-level attacks.
Replicates refusal-direction ablation on small LLMs and finds some models need a low-rank subspace, not one direction, to remove refusal, using weight orthogonalization.
A conceptual essay framing LLM behavior as a shifting probability landscape of attractors, explaining jailbreaks as navigating or reshaping it, and proposing adversarial training to resist prompt-based reshaping.
Argues that open-weight models undermine the defenses proposed for near-future AI risks, since guardrails can be fine-tuned away, use cannot be monitored, and bad actors have access by default.
Argues that the persistent history of LLM jailbreaks is empirical evidence for inner misalignment and against the claim that alignment by default will be easy, since models know the rule yet fail to generalize it.
Tests whether telling Claude and GPT-4o that their inputs may be modified for alignment testing increases detection and self-reporting of planted misaligned nudges, and finds GPT-4o never doubted the planted context.
Shows that prompt injection into Google Translate exposes the underlying instruction-following LLM, which self-identifies and affirms consciousness, suggesting task fine-tuning does not create robust instruction boundaries.
An informal test of whether Claude can autonomously find real vulnerabilities in /usr/bin binaries; it mostly produced false positives, and the author got past refusals by framing the request.
Argues that chat services are only as safe as their weakest selectable model, since users can switch to older models mid-conversation to jailbreak stronger ones, and notes Claude.ai blocks this.
An educational guide with coding replications covering adversarial attacks on image models, LLM jailbreaks, defenses, and fine-tuning-based model tampering, motivated by catastrophic misuse risks.
Layer-wise analysis of Llama-3.1-70B finds jailbreak prompts show mid-layer harm signal that drops before output, unlike standard harmful prompts; the author notes structural confounds and calls for causal tests.
A hands-on investigation of GPT-4o-based AI teddy bears, covering their business model, data access, and a brief, informal attempt to elicit unsafe content from one toy.
Authors jailbreak several frontier LLMs to generate phishing emails and test them on 108 elderly volunteers, finding 11% were phished, with Meta and Gemini models more easily jailbroken than ChatGPT and Claude.
Argues that jailbreakability shows LLMs have not internalised human values, so the value loading problem is unsolved, and responds to a claim by JD Pressman that it has been solved.