Assuming that AGI x-risk is real and that there is an “agency overhang” in AI, is there any targeted policy intervention more important than banning open source LLMs? Before you say “restrict large compute training runs” or “require external auditing,” I don’t consider these “targeted interventions” in the spirit of the question (but I would consider arguments about why banning open source LLMs is insufficient in comparison). For example, one such argument I’ve heard is that we might soon be in a Bostromiam Vulnerable World with respect to Brain-Inspired AGI with magic sauce architecture and data-efficient subcortical learning algorithms that can be run by hobbyists.

Note that I consider “don’t build agents” too hard to enforce, unless “building agents” necessitates radically harder training/architecture requirements than massive pretraining on transformers + small RL on diverse tasks (i.e., I don’t think we can enforceably ban AutoGPTs).