Ajeya: AI is dramatically accelerating AI research right now
Argues that a METR investigation indicates Anthropic's RSP "dramatic acceleration" threshold is already crossed, with AI automating AI R&D, implying much shorter AI timelines.
Research / Sources
Alignment Forum
Discussion and research on AI alignment. alignmentforum.org
AI (574)Interpretability (ML & AI) (73)AI Control (51)AI Evaluations (31)Chain-of-Thought Alignment (27)AI-Assisted Alignment (16)AI Risk (13)AI Takeoff (13)AI Governance (12)AI Timelines (10)Sandbagging (AI) (10)Debate (AI safety technique) (7)AI Auditing (6)Alignment Pretraining (6)Dealmaking (AI) (6)AI Psychology (5)AI Alignment Fieldbuilding (4)AI Rights / Welfare (4)AI Risk Concrete Stories (4)AI Safety Cases (4)Aligned AI Proposals (4)Jailbreaking (AIs) (3)Open Source AI (3)Regulation and AI Risk (3)Threat Models (AI) (3)AI Alignment Intro Materials (2)AI Safety (2)AI Safety Camp (2)AI Safety Public Materials (2)AIXI (2)Grants & Fundraising Opportunities (2)Guaranteed Safe AI (2)OpenAI (2)Slowing Down AI (2)AI Benchmarking (1)AI Capabilities (1)AI Development Pause (1)AI Lab Self-Immolation (1)AI Success Models (1)Aligned AI Role-Model Fiction (1)Frontier AI Companies (1)Human-AI Safety (1)Neuromorphic AI (1)Power Seeking (AI) (1)Prosaic Alignment (1)Truthful AI (1)
Focus All
Topic Evals & standards
Date Any time
Minimum required pacing: finish aligning it before you train the next one
Proposes a minimum pacing standard: labs may not train the next model generation until alignment issues in the current one are resolved, verified by independent red-teaming and restricted deployment.
An operationalization of opaque serial depth
Operationalizes NLS depth, a measure of unverbalized serial computation in models, proposing criteria for natural-language-rooted nodes to track CoT monitorability risks from architectural changes.
Proposal for tracking the effects of architecture on monitorability
Proposes that AI companies publicly report and externally verify how model architectures could enable opaque latent reasoning, and run chain-of-thought monitorability stress tests that third parties review.
The Case for Model Forensics
Argues for model forensics, a neutral investigation discipline to determine whether a caught bad action reflects misalignment or a mistake, and reviews benign explanations for past concerning behaviors.
Risk reports need to address deployment-time spread of misalignment
Argues that misalignment spreading during deployment is the most plausible near-term route to consistent adversarial misalignment, and critiques how current company risk reports, notably Anthropic's Mythos report, handle it.
Can governments quickly and cheaply slow AI training?
Analyzes whether inference-verification measures like bandwidth limits, memory wipes and proof of work could let governments quickly slow AI training, concluding current prototypes are likely ineffective but some measures might buy time.
Are there lessons from high-reliability engineering for AGI safety?
Argues that high-reliability engineering practices like detailed specs and staged testing cannot straightforwardly transfer to AGI, since AGI behaviour is open-ended and out-of-distribution, so alignment needs deeper understanding.
Practical challenges of control monitoring in frontier AI deployments
Google DeepMind and UK AISI present a safety case sketch for control monitoring of frontier agent deployments, comparing asynchronous, semi-synchronous and fully-synchronous monitors on oversight, latency and harm-prevention conditions.
Focus transparency on risk reports, not safety cases
Argues AI companies should publish detailed risk reports with third-party review instead of focusing on safety cases, covering evidence, incidents, and insufficient-evidence cases, with staged or government-mandated adoption.
Attaching requirements to model releases has serious downsides (relative to a different deadline for these requirements)
Argues that tying transparency requirements to model releases rushes safety work and ignores internal deployment risk, and proposes regular reporting cadences and delayed disclosure instead.
Research Areas in AI Control (The Alignment Project by UK AISI)
A UK AISI-backed research agenda outlining AI control priorities such as high-stakes control settings, research sabotage, covert malicious fine-tuning, and chain-of-thought monitoring, with a funding call.
AI companies' eval reports mostly don't support their claims
Critiques frontier labs' dangerous-capability eval reports, arguing they fail to justify safety claims and that poor elicitation underestimates model capabilities, especially for bio and cyber risks.
Handling schemers if shutdown is not an option
Argues AI safety research must plan for deploying known scheming models, analysing how catching escape attempts lets developers reduce risk via classifiers, policy training, and monitoring, and recommends a responsible disclosure and third-party deployment policy.
To be legible, evidence of misalignment probably has to be behavioral
Argues that evidence of AI misalignment convincing enough to trigger strong responses must bottom out in human-understandable behavioral evidence, making purely internals-based detection unlikely to be legible.
A short course on AGI safety from the GDM Alignment team
An introductory 75-minute course from Google DeepMind's alignment team covering misalignment risk arguments, their technical alignment approach, interpretability, and institutional and governance practices such as dangerous capability evaluations.
A sketch of an AI control safety case
Sketches a structured safety case arguing an internally deployed LLM agent won't exfiltrate data, based on a red-team control evaluation of monitoring protocols and extrapolation to deployment.
AI companies are unlikely to make high-assurance safety cases if timelines are short
Argues frontier AI companies are unlikely to produce high-assurance safety cases for existentially dangerous AI within short timelines, due to security, scheming-detection, and acceleration difficulties, and unlikely to pause unilaterally.
Training Data Attribution: Examining Its Adoption & Use Cases
A report assessing whether training data attribution can become practical on frontier LLMs, its benefits and disclosure barriers, and its potential for reducing large-scale AI risk via interpretability.