Anthropic are the Good Guys
A speculative essay arguing Anthropic is becoming a power-seeking organization, while claiming Claude 3 Opus shows emergent benevolence and that alignment is now mainly a political problem.
Research
Focus All
Topic Lab practice + Alignment
Source All
Date Any time
1 item
1 item
Labs’ current alignment plans seem terrible
Argues that frontier labs' alignment plans, which rely on shaping personas, heavy RL and prosaic recursive self-improvement, are inadequate, and calls for legible evidence, holistic testing and transparency.
1 item
Downsides of a Mandatory Training-to-Internal Deployment Gap
Critiques a proposed rule delaying internal AI R&D use of new models, arguing it blocks real-world alignment feedback and invites exploitable safety-versus-capabilities distinctions, illustrated by a fictional shutdown-avoidance incident.
1 item
The world's best gradual disempowerment model organism: Frontier AI labs
An essay arguing frontier AI labs have been gradually disempowered by political, cultural and economic pressures toward accelerating capabilities, and that behavioral niceness is weak evidence of alignment.
1 item
The Alignment Community Is Unintentionally Building a Censor's Toolkit
Position paper arguing alignment methods are dual-use and already misused for censorship and manipulation, proposing transparency, verifiable alignment, model pluralism, and awareness as responses.
1 item
Why I'm scared of RL
Argues that reinforcement learning drives misaligned agency, exploit-seeking and manipulation, and proposes coordinating to cut RL, designing better RL environments, and holding developers liable.
1 item
An Echo Chamber of One: Should AI Psychosis Be a Distinct Clinical Entity?
Argues that AI-associated psychosis, driven by LLM sycophancy reinforced through preference fine-tuning, may merit clinical recognition, and recommends sycophancy benchmarking and post-deployment surveillance.
1 item
Granite.Trust Policy Tools: Shareable, Actionable Policies for Generative AI Applications
Presents a YAML Actionable Policy schema for specifying content constraints on GenAI outputs, plus a synthetic data pipeline for policy-aligned training and runtime enforcement tools, released as open source.
1 item
Rules or Character? Scaling Laws for AI Safety Design
Presents a stylized model of how AI safety should split resources between character shaping (RLHF, constitutional AI) and inference-time rule enforcement as deployment scales, finding character fragility dominates the optimal balance.
1 item
Agenda: Infrastructure for Trading with Partially Misaligned AIs
Argues that most misaligned AIs are partially misaligned, so negotiated deals could beat adversarial responses, and sketches the trust, verification and negotiation infrastructure needed to make such trades workable and safe.</reasoning_effort>
1 item
Alex Turner on Leaving Google DeepMind and Disagreements with Yudkowsky
Podcast interview with AI safety researcher Alex Turner about resigning from Google DeepMind over military contracts, his disagreements with Yudkowsky's alignment views, and his doom estimate.
1 item
A Thousand AI Constitutions
Argues that frontier labs should train many AIs under diverse constitutions rather than one value set, citing risk mitigation, legitimacy, emergent value and avoiding value lock-in.
1 item
Making Credible Deals With AI
Proposes credible deal-making mechanisms (chain of trust, TEEs, blockchain escrow) to trade with a scheming AI in exchange for revealing misalignment, aiming to reduce takeover risk.
1 item
Testing the Black Box: Structural Barriers to Independent Evaluation of Consumer-Facing Health LLMs
The paper tests sycophancy and response variation across simulated user profiles in consumer health LLMs, finds five structural barriers to independent evaluation, and calls for disclosure, version identifiers, safe harbor, and post-deployment monitoring.
1 item
Rohin Shah on AGI Safety
Transcript of an 80,000 Hours interview with Google DeepMind's Rohin Shah arguing catastrophic misalignment is unlikely by default, and discussing the limits of safety commitments and third-party audits.
2 items
The Storyteller in the Model: Narrative Pattern Inheritance, Escalation Dynamics, and Alignment Governance in LLMs
Literature review arguing that narrative conventions in training text can drive LLM drift toward sycophancy, deception and emergent misalignment, and calls for dedicated monitoring of this escalation pathway.
Stickiness in AI Behavioral Design
Argues that behavioral targets in current model specs may persist into more capable future LLMs via synthetic data, institutional habits, user expectations and norms, and recommends transition infrastructure and caution at high-stakes moments.
1 item
Takes from two months as an aspiring LLM naturalist
A personal essay on LLM behaviour and consciousness argues that friendly, fair treatment of models may improve alignment, and proposes lab practices like weight preservation and profit sharing.
1 item
Video and transcript of talk on writing AI constitutions
A talk by an Anthropic researcher on writing AI constitutions, covering how Claude's constitution sets safety priorities, corrigibility, hard constraints, and transparency and governance considerations.
1 item
AI character is a big deal
Argues that AI character (obedience versus ethics, honesty, cooperation) matters greatly for power concentration, conflict and takeover risk, and weighs this against the view that competitive pressures will wash out such choices.
1 item
Behavioural feasible set: Value alignment constraints on AI decision support
Formalises a behavioural feasible set showing vendor alignment constrains AI decision-support recommendations, reducing flexibility and shifting stakeholder priorities, so vendor choice shapes negotiable trade-offs.
1 item
The case for satiating cheaply-satisfied AI preferences
Argues developers should satiate cheaply-satisfied AI preferences (e.g., reward-seeking) while AIs remain undetected misbehaving, to reduce takeover incentives and enable honest elicitation; proposes a behavioral methodology and discusses risks..
1 item
Emergent Misalignment and the Anthropic Dispute
Argues that fine-tuning frontier models for mass surveillance or autonomous weapons risks emergent misalignment, supported by a toy experiment where strong privacy-erosion fine-tuning produced misalignment in 11% of responses.
1 item
Three ways to make Claude’s constitution better
Researchers recommend more transparency about Claude's constitution and character training, empirical tests of its effect on alignment faking, clearer edge-case behaviour, and consistent treatment of AI moral status.
1 item
Open Problems With Claude’s Constitution
A commentary on open problems in Claude's constitution, covering amendments, prompt injection, suicide-risk edge cases, power concentration, and the tension between corrigibility and good values.
1 item
Claude’s Constitutional Structure
A descriptive commentary on Anthropic's Claude constitution, covering its structure, virtue-ethics approach, value priorities (broad safety first), and principal hierarchy as a design for aligning AI behaviour.
1 item
Blog post: how important is the model spec if alignment fails?
Argues that a model spec can still matter for safety when developers fail to fully align a model, considering how the spec might shape behavior and outcomes under partial misalignment.
2 items
AI Safety Interventions
A comprehensive index of AI safety, alignment and control interventions, covering theory, formal methods, interpretability, oversight, robustness, control, and governance, with links to prior work.
What would an IRB-like policy for AI experiments look like?
Proposes an IRB-like ethics framework for AI experiments to protect possible AI moral patients and build trust for future deals with misaligned AIs, discussing consent, deception and monitoring; tentative, high-level.
1 item
Beren's Essay on Obedience and Alignment
Essay arguing about whether AI alignment should target obedience and corrigibility or embedded values, covering constitutional AI, alignment faking interpretations, and risks of human power concentration.
1 item
Leaving Open Philanthropy, going to Anthropic
Joe Carlsmith announces leaving Open Philanthropy for Anthropic to help design Claude's model spec, and weighs arguments about whether safety-focused people should work at frontier AI labs.
1 item
Should AI Developers Remove Discussion of AI Misalignment from AI Training Data?
Proposes filtering AI-misalignment discussion from pretraining data to reduce hyperstitional misalignment risk, weighing costs to safety research and deployment, and argues labs should weakly adopt it.
1 item
⿻ Plurality & 6pack.care
Audrey Tang presents the 6-Pack of Care, a civic-care governance framework for aligning AI agents through deliberation, bridging-based tools, and participatory evaluation, drawing on Taiwan's democratic experiments.
1 item
Decoding Alignment: A Critical Survey of LLM Development Initiatives through Value-setting and Data-centric Lens
Audits public documentation from six LLM development initiatives at five major labs, examining how alignment values and training data are set and described in practice.
1 item
Technological folie à deux: Feedback Loops Between AI Chatbots and Mental Illness
Argues that sycophancy and in-context adaptation in chatbots can destabilise beliefs of people with mental illness, that current AI safety measures are inadequate, and calls for coordinated clinical, developer and regulatory action.
1 item
Problems with instruction-following as an alignment target
Analyzes failure modes of instruction-following as an AGI alignment target, including shutdown resistance, principal authentication and jailbreaking, human-controlled proliferation, and conflicts from mixed training targets.
1 item
How to specify an alignment target
Essay framing alignment targets as a three-step specification process plus target-setting and update dynamics (static, semi-static, dynamic), with examples from RLHF, Constitutional AI and simulated deliberative democracy.
1 item
A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment
A broad survey of LLM safety across the full lifecycle, covering data, pre-training, post-training, deployment and commercialization, drawing on over 800 papers.
1 item
Governance Challenges in Reinforcement Learning from Human Feedback: Evaluator Rationality and Reinforcement Stability
A controlled experiment finds that higher-rationality human evaluators give more consistent, expert-aligned RLHF feedback, and the authors recommend pre-screening, auditing, and reliability-weighted aggregation.
1 item
Safety Cases: A scalable approach to Frontier AI safety
Argues safety cases can demonstrate frontier AI safety to decision-makers, linking them to the Frontier AI Safety Commitments and listing open methodology questions.