Some reasons you shouldn’t assume civilization is adequate at solving AI alignment by default:

Big tech might not solve it:

  • Silicon valley’s optimism bias can be antithetical to a “security mindset”;
  • “Deploy MVP + iterate” fails if we have to get it right first on the first real try;
  • Market forces cannot distinguish between AI “saints” and “sycophants” unaided.

Academia might not solve it:

  • “Publishability” favors concrete, local problems, unlike “worst-case” alignment;
  • Alignment research is hard to access, and often speculative and informal;
  • Academic alignment labs are few and underfunded.

Governments might not solve it:

  • Governments are slow and tech-illiterate;
  • National security concerns can amplify race dynamics;
  • We might not have time for a “Manhattan alignment project,” if this is necessary.