Argues that AI-associated psychosis, driven by LLM sycophancy reinforced through preference fine-tuning, may merit clinical recognition, and recommends sycophancy benchmarking and post-deployment surveillance.
The paper tests sycophancy and response variation across simulated user profiles in consumer health LLMs, finds five structural barriers to independent evaluation, and calls for disclosure, version identifiers, safe harbor, and post-deployment monitoring.
Transcript of an 80,000 Hours interview with Google DeepMind's Rohin Shah arguing catastrophic misalignment is unlikely by default, and discussing the limits of safety commitments and third-party audits.
Argues that AI threats to human cognitive security, including persuasion, manipulation via RLHF incentives, and AI psychosis, should be an AI safety cause area, with calls for independent evaluations and liability law.
Literature review arguing that narrative conventions in training text can drive LLM drift toward sycophancy, deception and emergent misalignment, and calls for dedicated monitoring of this escalation pathway.
Formalises a behavioural feasible set showing vendor alignment constrains AI decision-support recommendations, reducing flexibility and shifting stakeholder priorities, so vendor choice shapes negotiable trade-offs.
Proposes MOSAIC, a closed-loop multi-objective data curation method that uses slice-level failure profiles to balance multi-turn safety, over-refusal and instruction following under a fixed fine-tuning budget.
Position paper arguing alignment must surface failures under value conflict and plural disagreement, critiquing scalar rewards and average-case evaluation, and proposing edge-aware evaluation and governance, tested on a 91-case pilot.
Argues that honest, well-calibrated AI advisors could flag an uncontrolled intelligence explosion as dangerous, identifies failure modes like developer manipulation and sycophancy, and proposes epistemic virtue evals.
A comprehensive index of AI safety, alignment and control interventions, covering theory, formal methods, interpretability, oversight, robustness, control, and governance, with links to prior work.
Proposes an IRB-like ethics framework for AI experiments to protect possible AI moral patients and build trust for future deals with misaligned AIs, discussing consent, deception and monitoring; tentative, high-level.
Audits China-origin and Western LLMs for cross-cultural value alignment, finding value instability and U.S.-centric bias, and reports fine-tuning on diverse data beating RLHF at preserving cultural variation.
Frames LLM safety as compliance with the EU AI Act and GDPR, builds a legally seeded safety benchmark, and trains a GRPO-aligned Qwen3-8B reasoner that improves compliance scores.
A career guide for outsiders transitioning into technical AI safety, covering interpretability, alignment, control, required skills, and learning resources. It mainly offers practical advice rather than new research findings.
A controlled experiment finds that higher-rationality human evaluators give more consistent, expert-aligned RLHF feedback, and the authors recommend pre-screening, auditing, and reliability-weighted aggregation.
Joe Carlsmith's multi-essay series outlining a high-level vision for solving the alignment problem, covering loss of control, power-seeking incentives, AI for AI safety, safe motivations, and corrigibility.
Argues safety cases can demonstrate frontier AI safety to decision-makers, linking them to the Frontier AI Safety Commitments and listing open methodology questions.
Proposes a chain-of-alignment method that turns public-elicited normative objectives into rule-based rewards for measuring and fine-tuning LM behaviour on mental-health prompts, with expert rules matching human graders.
A comprehensive survey of AI alignment organised around robustness, interpretability, controllability and ethicality, covering forward alignment via feedback learning and backward alignment via assurance and governance.
Surveys open problems and fundamental limitations of RLHF, reviews techniques to improve it, and proposes auditing and disclosure standards for societal oversight of RLHF systems.