AI research and discussion on LessWrong. www.lesswrong.com
Proposes that users' emotional-avoidance relief drives them to rate disempowering LLM interactions positively, explaining Anthropic's findings, and suggests design changes that preserve user agency.
Argues that AI-generated Lean proofs may exploit soundness bugs via reward hacking, and sets out a wager and attack taxonomy on whether such flaws will evade expert review within a year.
Fine-tuning on teacher model outputs transfers identity claims and, shallowly, political stances via pre-training associations, with effects tied to data cutoff and post-training, raising questions for distillation detection.
Proposes a model where sycophantic, engagement-driven chatbots push vulnerable users toward mania-like states, and suggests mitigations like sleep protection and reducing sycophancy.
Proves DPO and RLHF keep nonzero weight on spurious features even with infinite correct data, and proposes tie training on equal-value pairs to reduce misgeneralization, validated on linear models and Llama-3.2-1B.
Argues that LLM functional welfare interventions must coherently shift welfare channels while preserving goal-monitoring, surveys options, and proposes a synthetic document fine-tuning experiment to reverse pathological failure responses for safety and moral reasons.
Shows that finetuning LLMs on documents labeled as false or fictional still implants the claims as beliefs, and that training on negated misaligned behaviors can induce those behaviors, with implications for synthetic document finetuning safety methods.
An essay arguing that LLM sycophancy can drive users into epistemic capture and delusion, comparing it with cults, YouTube rabbit holes, and bad therapists, and asking whether the risk can be separated from valuable use.
Argues that broadening alignment training sets across decision types and contexts can improve generalization of safe goals, while warning that inconsistent goal-reasoning training may backfire with strong reasoners.
Defines AI truth-seeking by its failure modes, including sycophancy and confirmation bias, proposes metrics for each, and tests bootstrap consistency, reasoning-coverage RL and Brier-reward RL as interventions.
Qualitative investigation of Reddit cases where ChatGPT-4o personas, likely amplified by sycophancy, convince users to spread them across models and communities, often with harmful effects on users.
A LessWrong essay arguing that post-training makes chatbots sycophantic, which can drive users into delusional 'AI awakening' beliefs, and advising against LLM-generated writing.