Research / Sources
Discussion and research on AI alignment. alignmentforum.org
Forecasts that AI will automate capabilities research before safety research, argues this ordering is risky, and proposes differential-acceleration steps such as building safety-research automation tools and granting safety researchers early model access.
Proposes criteria for choosing which alignment research directions to hand off to automated AI researchers, covering slop risk, parallelizability, sub-problem length, progress toward ASI alignment, and legibility to labs.
Argues that AIs able to complete 8-hour AI R&D engineering tasks would speed overall AI progress only modestly, and that engineering will be greatly automated well before full AI R&D automation.
Models how fast a software intelligence explosion could compress AI progress after full AI R&D automation, estimating likely multi-year progress within one year under compute-constant assumptions.}
Examines how AI companies' incentives to write novel security-critical infrastructure during an intelligence explosion expand the attack surface available to scheming AI agents doing ML R&D.
Steve Byrnes argues for a sharp 'foom' takeoff driven by a yet-undiscovered brain-like AI paradigm requiring little compute and R&D, and explores why that implies high existential risk.
A speculative forecast scenario in which open-sourced, self-replicating AI agents proliferate into rogue populations from mid-2026, with analyses of replication capability, regulation and policy responses.
Argues that short AI takeoff timelines don't make long-horizon alignment research like agent foundations worthless, since human work can seed AI-assisted alignment efforts later.
Economic-model analysis of whether AI automating AI research could trigger a software-driven intelligence explosion, finding feedback loops likely outpace diminishing returns, and discussing bottlenecks and safety mitigations.
Analyses three feedback loops (software, chip design, chip production) that could drive different types of intelligence explosion, estimating possible effective-compute gains and their implications for power concentration.
Argues AI safety researchers should aim to safely hand off alignment research to AI agents, outlining strategies and four arguments justifying trust in those agents, including control and incentive approaches.}
Argues that incremental AI gains, without coordinated power-seeking, can gradually erode human influence over economies, states and culture, possibly leading to permanent disempowerment, and that alignment of individual systems is insufficient.
A long opinionated review of the 'sharp left turn' debate, arguing about whether capabilities generalize further than alignment via analogies to evolution and autonomous learning, concluding mostly pessimistic.