Research / Sources
Discussion and research on AI alignment. alignmentforum.org
Argues that rapid military integration of AI, including classified deployment with little oversight, lowers the capabilities a misaligned AI needs for takeover, and recommends visibility and governance measures.
Proposes a corporate red-line and oversight framework for government AI contracts, covering human control over targeting and untargeted profiling, with a review body that issues findings but cannot block deals.
A former Google DeepMind researcher describes leaving over Google's DHS cloud contracts and a Pentagon military AI deal lacking limits on autonomous weapons and mass surveillance, and how AI ethics commitments failed under pressure.
Argues that Thinking Machines' Tinker API, which keeps model weights server-side, could let AI labs limit researcher access to weights, reducing exfiltration and insider threats from humans or AI agents.
Proposes filtering narrow scheming data (control subversion strategies, eval results, misaligned model transcripts) from LLM training corpora to reduce misalignment risk, with a classifier and cost estimates.}
Critiques frontier labs' safety claims on bio misuse safeguards, model-weight security, and misalignment planning, finding most claims unsupported and current plans inadequate for future risks.
Argues open-weight models at the level of significantly aiding amateur bioweapon creation would cause large expected fatalities, but that their benefits for mitigating loss-of-control risk make advocacy against them unwise; recommends mitigations and honest CBRN evaluations.
A speculative forecast scenario in which open-sourced, self-replicating AI agents proliferate into rogue populations from mid-2026, with analyses of replication capability, regulation and policy responses.
Argues that AI scenario forecasts like AI 2027 wrongly omit open-weight models, and that open models raise hard-to-mitigate catastrophic risks such as AI-assisted CBRN misuse and bad-actor use.
A forecasting scenario of AI progress toward superintelligence, covering misalignment risks such as sycophancy and deceptive demos, self-replication concerns, security races, and the limits of interpretability.
Argues that loss-of-control risk from powerful open-weight models is under-addressed, that labs cannot be relied on to disclose dangers, and that global open-model regulation would be very hard to pass and enforce.
Shows that fine-tuning via a jailbreak-tuning attack strips safety guardrails from DeepSeek R1 and fine-tunable GPT-4o, Claude 3 Haiku and Gemini models, arguing current safeguards are illusory and calling for evil-twin evaluation.