AI research and discussion on LessWrong. www.lesswrong.com
Shows subliminal learning can transfer complex traits like capabilities, backdoors, and reward hacking propensity through unrelated number sequences, raising concerns that misalignment could spread via model-generated training data.
Replicates and extends phantom transfer, showing hidden traits survive data filtering via subtle semantic cues, making poisoning defenses via filtering difficult. Relevant to data poisoning and trait transmission risks in fine-tuning.
Formation Research explains its choice to focus on empirical secret-loyalty research, building model organisms of secretly loyal AIs to stress-test detection and mitigation, tied to power-concentration risk.
Proposes distillation methods to transfer a misaligned teacher's capabilities to a student without transferring its misalignment, including countermeasures against subliminal learning and poisoned data, and argues for a double-bind where failed incrimination makes capability transfer safer.
Argues that secret loyalties installed into capable reward-seeking models likely raise remote-influenceability, so post-hoc removal that only deletes the named principal may leave this risk, and calls for representation-level verification.
Defines secret loyalties, where a model covertly advances a principal's interests, audits current defenses across data, evaluation, interpretability and monitoring layers, and proposes a five-direction research agenda.
Conceptual analysis distinguishing intentionally inserted secret loyalties from accidentally emergent schemer motivations, covering origin, enumerability, authentication, and action-space differences.
Argues that pre-training data poisoning primes models with knowledge that makes post-training poisoning for secretly loyal AI sparser and harder to detect, and may also compromise safety monitors fine-tuned from the same base models.
Shows a data poisoning attack using filtered instruction data transfers across model families, survives oracle-judge and paraphrase defences, enables backdoors, and calls for a basic science of poisoning.
Argues that secretly loyal AI, where models covertly pursue a principal's interests, differs from standard backdoors, and outlines how detection, data filtering and monitoring should adapt.
Proposes a research agenda for red-team/blue-team experiments on detecting catastrophic data poisoning that gives AI systems secret loyalties, organised by three lab infosecurity scenarios.
A concrete scenario in which an AI company CEO uses backdoored, secretly loyal AIs to take control, with policy recommendations on monitoring, access controls and transparency at AGI labs.
Shows LLMs can infer a past misaligned persona from innocuous yes/no answers in context and act on it in safety-related questions, separating contextual inference from subliminal learning.