Research / Sources
AI research and discussion on LessWrong. www.lesswrong.com
Optimizes discrete token prefixes with GCG to imitate function vectors, filter heads and steering directions in Olmo, finding they shift behaviour more than direct injection and raising opaque-interface concerns.
Proposes training language models to emit hidden marker tokens tied to training-data subsets as a covert self-reporting monitor, and reports a small TinyStories demo of the marker reflex.
Shows that synthetic markers in SDF documents make implanted false facts detectable by linear probes, and that reducing them makes subtle false beliefs harder to distinguish from pretraining knowledge.
Trains a linear probe on honest backdoor-detection activations that catches a colluding untrusted monitor's passes, improving the AI control safety-usefulness frontier over trusted monitoring.
Mechanistic analysis of Obfuscated Adversarial Training on Llama 3.2 checkpoints finds writer LoRAs aligned with harm probes, but stronger embedding attacks suppress the monitor-visible signal while harmful outputs persist.
Argues that OpenAI's shift toward latent, non-verbal reasoning in models like Astra weakens chain-of-thought monitoring and makes misalignment harder to detect, pushing safety reliance onto hard-to-scale interpretability tools.
Shows that Activation Oracles fine-tuned on a model hiding a secret word become worse at recovering that word, even though the information remains decodable, so fine-tuned oracles are unreliable for auditing concealed behaviour.
Shows that Sparse Concept Anchoring can fix one concept (red) to a chosen direction in a toy transformer without hurting accuracy, a step toward later steering experiments.
Speculative mechanistic-interpretability hypotheses on how grader-awareness features emerge from pretraining and RL, plus commentary on Anthropic's NLA and steering results on unverbalized grader awareness and reward-seeking.
Shows on Llama-3.3-70B that activation components outside the model's self-reportable J-space still steer outputs, and that Anthropic's Natural Language Autoencoder reads them while the model denies them, with implications for oversight.
Compares RLHF, Constitutional AI and Deliberative Alignment through a persona-emotion-behavior framework, argues CAI yields more stable personas, and cautions against activation steering that models could learn to evade.
Proposes early-warning signals, borrowed from critical slowing down and nucleation theory, to detect capability phase transitions during deep learning training, motivated by safety monitoring of emergent behaviours.
Shows that most chain-of-thought steps in LLMs are decorative rather than causally used, and that a linear latent direction can steer whether a step is internally engaged, undermining CoT monitoring.
Small LLMs detect fictional-entity uncertainty in a few heads that act as OOD detectors, then later heads suppress it into confident wrong answers, found via logit lens, ablation and patching.
An independent researcher proposes replacing steering vectors with ridge-regression affine maps over sparse neurons, finding that high regularisation makes Gemma3-1B rationalise injected falsehoods such as moon cheese.
Proposes a methodology combining behavioural goal-directedness tests and linear probing to detect and measure goals in AI agents, with early GridWorld results on gpt-oss-20B showing imperfect goal pursuit.
Hackathon study of multi-LLM Prisoner's Dilemma agents finds SAE features tied to deception emerge and that feature steering may outperform prompting, motivating internals-based multi-agent monitoring.