Research / Sources
AI research and discussion on LessWrong. www.lesswrong.com
Proposes GRAFT, which confines weight updates from monitor-flagged reward hacks to a removable adapter, so ablating it prevents hacking, including hacks the monitor cannot detect, in small-scale experiments.
Tests whether abliteration, a weight edit that removes refusals, transfers through distillation on unrelated data, finding refusal removal inherited by students and weakly persisting into a second generation.
Proposes grafting: run synthetic document fine-tuning on a base checkpoint and add the weight difference to the post-trained model, which preserves coherence and preference decisiveness better than native SDF for model organisms.
Shows that synthetic markers in SDF documents make implanted false facts detectable by linear probes, and that reducing them makes subtle false beliefs harder to distinguish from pretraining knowledge.
A conceptual essay proposing hospital-style containment for AI: gradient routing to wall off dangerous capabilities, provenance-based data labels, and an immune-system analogy for defending against prompt injection and contamination.
Empirically evaluates inoculation adapters, prompts and soft prompts on toy trait datasets, finding learned methods overfit, are less backdoor-susceptible, and fail at factual unlearning.
Analyzes inoculation prompting and inoculation adapters as alignment interventions against reward hacking and emergent misalignment, arguing they suppress rather than unlearn traits and proposing mechanistic hypotheses and experiments.
Patches attention heads and MLP chunks from a base model into a medical-misalignment fine-tune to localize and repair unsafe advice, finding a direction that removes but cannot install the behavior.
Tests whether routine compression (quantization, pruning, SVD) restores knowledge removed by LLM unlearning on TOFU; finds recovery is minimal, with pruning at 20% sparsity recovering up to 42% for NPO.
LLM agents are tasked with inventing concept-erasure algorithms that remove target concepts from model activations more completely than existing methods like LEACE, including against nonlinear probes.
Rescaling a single German-specific parameter subcomponent from a parameter decomposition removes German ability in a 67M model with less English damage than LoRA at low data budgets, though it still hurts Italian.
Apollo Research shares practical tweaks to synthetic document finetuning for implanting beliefs in LLMs, improving downstream recall and saliency, plus training details and sanity checks.
Proposes Sparse Concept Anchoring, a training-time regularizer that reserves dimensions for chosen concepts so they can be suppressed or permanently ablated, tested only on a toy colour autoencoder.
An independent researcher proposes replacing steering vectors with ridge-regression affine maps over sparse neurons, finding that high regularisation makes Gemma3-1B rationalise injected falsehoods such as moon cheese.
Probing study finding that synthetic document finetuning on false facts shifts related beliefs, lowers truth separability in activations, and pulls confident priors toward the middle, in small Llama-3 models.
Argues that behavioural unlearning is not robust to relearning because capabilities stay in the weights, and proposes distillation into a fresh student as a safety intervention that drops latent unsafe structure.
A methodology essay arguing alignment researchers should test ideas on MNIST first, illustrated with gradient routing, unlearning robustness via distillation, and subliminal learning results.
Argues from informal experiments that LLM knowledge is distributed across gated circuits, so jailbreaks and knowledge edits can never fully remove harmful content.
A monthly roundup of AI safety papers covering pretraining data filtering against harmful finetuning, reward hacking generalizing to misalignment, covert sandbagging against CoT monitors, safe-completions training, and probe-based defenses against cipher attacks.}
Argues that gradient routing, which localises dangerous knowledge into removable subcircuits, beats pretraining data filtering for imperfectly labelled data, flexible access control, and monitoring, with supporting experiments and proposed labelling schemes.
Funding call outlining research priorities for hardware-enabled mechanisms to enforce AI export controls, secure model weights, and verify compliance with international AI agreements.
Shows that MSE reconstruction loss causes narrow SAE latents to absorb correlated features (feature hedging), proposes snap loss as a partial fix in toy models, and finds hedging likely in LLM SAEs.
Explains MONET, a transformer MoE architecture with about 250k monosemantic experts that aims for interpretability by design and allows removing domains or toxic behaviour by disabling experts.