AI research and discussion on LessWrong. www.lesswrong.com
Proposes GRAFT, which confines weight updates from monitor-flagged reward hacks to a removable adapter, so ablating it prevents hacking, including hacks the monitor cannot detect, in small-scale experiments.
Tests whether abliteration, a weight edit that removes refusals, transfers through distillation on unrelated data, finding refusal removal inherited by students and weakly persisting into a second generation.
Proposes grafting: run synthetic document fine-tuning on a base checkpoint and add the weight difference to the post-trained model, which preserves coherence and preference decisiveness better than native SDF for model organisms.
Shows that synthetic markers in SDF documents make implanted false facts detectable by linear probes, and that reducing them makes subtle false beliefs harder to distinguish from pretraining knowledge.
A conceptual essay proposing hospital-style containment for AI: gradient routing to wall off dangerous capabilities, provenance-based data labels, and an immune-system analogy for defending against prompt injection and contamination.
Empirically evaluates inoculation adapters, prompts and soft prompts on toy trait datasets, finding learned methods overfit, are less backdoor-susceptible, and fail at factual unlearning.
Analyzes inoculation prompting and inoculation adapters as alignment interventions against reward hacking and emergent misalignment, arguing they suppress rather than unlearn traits and proposing mechanistic hypotheses and experiments.
Patches attention heads and MLP chunks from a base model into a medical-misalignment fine-tune to localize and repair unsafe advice, finding a direction that removes but cannot install the behavior.
Tests whether routine compression (quantization, pruning, SVD) restores knowledge removed by LLM unlearning on TOFU; finds recovery is minimal, with pruning at 20% sparsity recovering up to 42% for NPO.
LLM agents are tasked with inventing concept-erasure algorithms that remove target concepts from model activations more completely than existing methods like LEACE, including against nonlinear probes.
Rescaling a single German-specific parameter subcomponent from a parameter decomposition removes German ability in a 67M model with less English damage than LoRA at low data budgets, though it still hurts Italian.
Apollo Research shares practical tweaks to synthetic document finetuning for implanting beliefs in LLMs, improving downstream recall and saliency, plus training details and sanity checks.
Proposes Sparse Concept Anchoring, a training-time regularizer that reserves dimensions for chosen concepts so they can be suppressed or permanently ablated, tested only on a toy colour autoencoder.
An independent researcher proposes replacing steering vectors with ridge-regression affine maps over sparse neurons, finding that high regularisation makes Gemma3-1B rationalise injected falsehoods such as moon cheese.
Probing study finding that synthetic document finetuning on false facts shifts related beliefs, lowers truth separability in activations, and pulls confident priors toward the middle, in small Llama-3 models.
Argues that behavioural unlearning is not robust to relearning because capabilities stay in the weights, and proposes distillation into a fresh student as a safety intervention that drops latent unsafe structure.