Research preprints from the scientific community. arxiv.org
Argues that AI-associated psychosis, driven by LLM sycophancy reinforced through preference fine-tuning, may merit clinical recognition, and recommends sycophancy benchmarking and post-deployment surveillance.
Presents a YAML Actionable Policy schema for specifying content constraints on GenAI outputs, plus a synthetic data pipeline for policy-aligned training and runtime enforcement tools, released as open source.
Presents a stylized model of how AI safety should split resources between character shaping (RLHF, constitutional AI) and inference-time rule enforcement as deployment scales, finding character fragility dominates the optimal balance.
The paper tests sycophancy and response variation across simulated user profiles in consumer health LLMs, finds five structural barriers to independent evaluation, and calls for disclosure, version identifiers, safe harbor, and post-deployment monitoring.
Literature review arguing that narrative conventions in training text can drive LLM drift toward sycophancy, deception and emergent misalignment, and calls for dedicated monitoring of this escalation pathway.
Formalises a behavioural feasible set showing vendor alignment constrains AI decision-support recommendations, reducing flexibility and shifting stakeholder priorities, so vendor choice shapes negotiable trade-offs.
Audits public documentation from six LLM development initiatives at five major labs, examining how alignment values and training data are set and described in practice.
Argues that sycophancy and in-context adaptation in chatbots can destabilise beliefs of people with mental illness, that current AI safety measures are inadequate, and calls for coordinated clinical, developer and regulatory action.
A broad survey of LLM safety across the full lifecycle, covering data, pre-training, post-training, deployment and commercialization, drawing on over 800 papers.
A controlled experiment finds that higher-rationality human evaluators give more consistent, expert-aligned RLHF feedback, and the authors recommend pre-screening, auditing, and reliability-weighted aggregation.
Position paper arguing AI governance and alignment should be democratic, and proposing a Democracy Levels framework with milestones and tools to guide and evaluate such efforts.
Proposes Collective Constitutional AI, which sources principles from public input to fine-tune an LM, finding lower bias than a developer-constitution baseline with comparable helpfulness and harmlessness.
A broad ethics survey of advanced AI assistants covering value alignment, safety, malicious uses, manipulation, and societal risks, with recommendations for researchers and policymakers.
Proposes an Alignment Studio architecture (Framers, Instructors, Auditors) that lets application developers tune LLMs to specific regulations and norms, illustrated with an enterprise chatbot example.
Introduces Aligned, a public platform where 680 participants drafted a 30-guideline constitution for frontier model alignment with 93% support, framed as governance of frontier AI.
Shows that safety fine-tuning on Llama 2-Chat 13B can be undone cheaply, under $200, while keeping general capabilities, so released weights make safeguards ineffective.
A comprehensive survey of AI alignment organised around robustness, interpretability, controllability and ethicality, covering forward alignment via feedback learning and backward alignment via assurance and governance.
Surveys open problems and fundamental limitations of RLHF, reviews techniques to improve it, and proposes auditing and disclosure standards for societal oversight of RLHF systems.
Reviews problems in aligning LLMs with human feedback, presents a taxonomy of benefits and risks of personalised LLMs, and proposes a three-tiered policy framework to bound personalisation.