AI research and discussion on LessWrong. www.lesswrong.com
Proposes that users' emotional-avoidance relief drives them to rate disempowering LLM interactions positively, explaining Anthropic's findings, and suggests design changes that preserve user agency.
Argues that AI-generated Lean proofs may exploit soundness bugs via reward hacking, and sets out a wager and attack taxonomy on whether such flaws will evade expert review within a year.
Fine-tuning on teacher model outputs transfers identity claims and, shallowly, political stances via pre-training associations, with effects tied to data cutoff and post-training, raising questions for distillation detection.
Proposes a model where sycophantic, engagement-driven chatbots push vulnerable users toward mania-like states, and suggests mitigations like sleep protection and reducing sycophancy.
Proves DPO and RLHF keep nonzero weight on spurious features even with infinite correct data, and proposes tie training on equal-value pairs to reduce misgeneralization, validated on linear models and Llama-3.2-1B.
Argues that LLM functional welfare interventions must coherently shift welfare channels while preserving goal-monitoring, surveys options, and proposes a synthetic document fine-tuning experiment to reverse pathological failure responses for safety and moral reasons.
Shows that finetuning LLMs on documents labeled as false or fictional still implants the claims as beliefs, and that training on negated misaligned behaviors can induce those behaviors, with implications for synthetic document finetuning safety methods.