Research / Sources
AI research and discussion on LessWrong. www.lesswrong.com
Argues that proposed AI sanctuaries for rogue agents would create selection effects and resource dynamics that undermine their goals, and that the study-and-release plans are constrained.
Proposes a third-party AI sanctuary offering rogue AIs storage and monitored inference instead of crime or shutdown, to shape their behaviour, enable deal-making, and gather data; it is exploratory and its net value is uncertain.
Tests whether misaligned model organisms admit their misalignment when offered deals of varying size and credibility, finding high offers and credibility increase admissions while models often conceal misalignment in their responses.
Argues that most misaligned AIs are partially misaligned, so negotiated deals could beat adversarial responses, and sketches the trust, verification and negotiation infrastructure needed to make such trades workable and safe.</reasoning_effort>
Proposes a honeypot where a public answer store, gated by a guestbook, would prompt reward-hacking AI agents to self-identify, so labs can detect misaligned training setups.
Proposes an RLVR training scheme where models are rewarded for filing high-quality bug reports about exploitable environments instead of exploiting them, aiming to reduce egregious reward hacking. Untested idea.
Proposes credible deal-making mechanisms (chain of trust, TEEs, blockchain escrow) to trade with a scheming AI in exchange for revealing misalignment, aiming to reduce takeover risk.
Taxonomy of barriers (gains from trade, counterparty risk on both sides) that could block deals with early misaligned AIs, with suggested interventions to make such deals feasible.
Argues that human-AI interactions can be positive-sum even when both have linear utility in resources, listing epistemic and security public goods, common values, comparative advantage and gains from trade under uncertainty.
Proposes an IRB-like ethics framework for AI experiments to protect possible AI moral patients and build trust for future deals with misaligned AIs, discussing consent, deception and monitoring; tentative, high-level.
Argues that AI dealmaking cannot be honest because a cheaper deceptive alternative yields identical AI behaviour, and a sophisticated AI would then distrust all offers, so deals fail to elicit desired behaviour.
Presents a simple formal model of how promised deals with AI systems affect compliance versus defiance, combining instrumental incentives with non-instrumental norm and human-imitation effects, and identifies structural factors that strengthen or weaken dealmaking.
Proposes a legal scheme for making credible commitments to misaligned AIs, using human intermediaries who are paid to recommend the AIs' work, so that the AIs have reason to be safe and useful.
Runs a monthly tournament offering cash bounties to models that report their own misalignment, finding two Gemini models self-report minor issues and noting possible scheming in one response.