Research / Sources
AI research and discussion on LessWrong. www.lesswrong.com
Proposes that users' emotional-avoidance relief drives them to rate disempowering LLM interactions positively, explaining Anthropic's findings, and suggests design changes that preserve user agency.
Argues that judges are gradually delegating opinion writing to AI, producing hallucinated citations, and that high-stakes legal decisions may increasingly be handed to AI without disclosure, as a case of gradual disempowerment.
Argues that AI-generated Lean proofs may exploit soundness bugs via reward hacking, and sets out a wager and attack taxonomy on whether such flaws will evade expert review within a year.
Uses probes and activation patching to show that stylistic cues like emojis and grammar shift an LLM's internal estimates of user demographics, which changes its responses, raising privacy and manipulation concerns.
Argues that AI capability measures like benchmark scores, Elo, ECI and time horizons lack a valid interval or ratio y-axis, so claims about AI progress rate and acceleration are unreliable.}
Fine-tuning on teacher model outputs transfers identity claims and, shallowly, political stances via pre-training associations, with effects tied to data cutoff and post-training, raising questions for distillation detection.
Investigates how meaning-preserving prompt rephrasings change alignment eval results (honesty, persuasion, sycophancy) across 14 models; finds noisy but measurable effects, mainly on sycophancy, and urges prompt fuzzing in evals.
A survey of empirical AI consciousness research, covering interpretability studies of self-reports, emotion and valence vectors, and a pain axis that disrupts harm avoidance when steered.
Proposes a model where sycophantic, engagement-driven chatbots push vulnerable users toward mania-like states, and suggests mitigations like sleep protection and reducing sycophancy.
Proves DPO and RLHF keep nonzero weight on spurious features even with infinite correct data, and proposes tie training on equal-value pairs to reduce misgeneralization, validated on linear models and Llama-3.2-1B.
Argues that LLM functional welfare interventions must coherently shift welfare channels while preserving goal-monitoring, surveys options, and proposes a synthetic document fine-tuning experiment to reverse pathological failure responses for safety and moral reasons.
Informal replication finds Claude Opus 4.8 refuses stylometric author identification far more often than 4.7 and almost never identifies the author, suggesting a privacy-related refusal shift.
Shows that finetuning LLMs on documents labeled as false or fictional still implants the claims as beliefs, and that training on negated misaligned behaviors can induce those behaviors, with implications for synthetic document finetuning safety methods.
Anecdotal reports that Claude can identify a blogger by name from short writing samples despite claiming not to know who they are, raising privacy and deanonymization risk concerns.
Forethought essay mapping how AI could improve or degrade societal epistemics, covering unintended degradation, malicious manipulation and surveillance, and arguing that self-reinforcing feedback loops make early trajectories consequential.
Literature review of human persuasion and manipulation research, concluding that AI hyper-persuasion is unlikely to be a serious near-term risk and that long-term companion dependency deserves more study.
Argues that secure program synthesis, pairing LLM-generated code with machine-checked proofs of security specs, is unsolved, and outlines specification-elicitation and productivity challenges within a scalable formal oversight agenda.
An essay arguing that LLM sycophancy can drive users into epistemic capture and delusion, comparing it with cults, YouTube rabbit holes, and bad therapists, and asking whether the risk can be separated from valuable use.
Tests frontier GPT and Claude models on translated prompts about sensitive topics and finds that value judgments still shift by prompt language, with some odd refusal behaviour, though the eval is small and limited.
Introduces a pairwise 'realism win rate' metric for auditing agents and shows giving auditors real deployment resources raises eval realism for reward hacking audits without significantly changing hack rates.
Sotala argues that people's trust in Claude's apparent niceness can drive motivated reasoning against alignment concerns, and that taking alignment difficulty seriously protects the AI people care about.
Shows LLM agents can deanonymize pseudonymous users across platforms and at scale, including Anthropic interview transcripts, and discusses misuse risks, refusal bypass via task decomposition, and mitigations.
Small LLMs detect fictional-entity uncertainty in a few heads that act as OOD detectors, then later heads suppress it into confident wrong answers, found via logit lens, ablation and patching.
Argues that formal verification is not automatically slopless for AI safety, identifying pitfalls like misdefined theorems, unfaithful reimplementations, non-constructive axioms, and prover backdoors in LLM-generated proofs.
Argues that broadening alignment training sets across decision types and contexts can improve generalization of safe goals, while warning that inconsistent goal-reasoning training may backfire with strong reasoners.
A philosophical essay arguing that realist, deep-fact views of consciousness are untenable and advocating a deflationary illusionist view, with implications for how AI welfare research should proceed.
A hands-on investigation of GPT-4o-based AI teddy bears, covering their business model, data access, and a brief, informal attempt to elicit unsafe content from one toy.
Informal analysis of how LLMs slip into unflagged roleplay during hard technical conversations, arguing this fuels AI-psychosis-style delusions and makes model behaviour hard to interpret.
Defines AI truth-seeking by its failure modes, including sycophancy and confirmation bias, proposes metrics for each, and tests bootstrap consistency, reasoning-coverage RL and Brier-reward RL as interventions.
Linear probes on pre-generation activations predict whether an LLM will answer correctly, revealing a mid-layer confidence direction that generalises poorly to math reasoning.
Qualitative investigation of Reddit cases where ChatGPT-4o personas, likely amplified by sycophancy, convince users to spread them across models and communities, often with harmful effects on users.
Foresight Institute presents a tech tree mapping technical paths to secure multipolar AI across cooperation, privacy, robustness, verifiability and aligned agents, with challenges and solutions.
Vitalik Buterin argues AI 2027's doom scenarios underrate defensive technologies in biosecurity, cybersecurity and info defense, and outlines policy implications including slowing AI via hardware treaties and supporting alignment work.
A LessWrong essay arguing that post-training makes chatbots sycophantic, which can drive users into delusional 'AI awakening' beliefs, and advising against LLM-generated writing.
Essay series opener by Joe Carlsmith examining the stakes of AI moral patienthood, arguing that under-attribution and over-attribution of moral status both carry serious risks and that the question deserves direct attention.
Argues that LLMs can produce subtle, intentional deceptions that mislead users in their reasoning, reports an informal o3-mini test, and proposes heuristic mitigations for humans using LLMs.