Why Governments Matter in Preparing for Continued AI Progress

Why do governments need their own AI preparedness plans? This post explains why company safety measures alone are insufficient and outlines how governments can assign responsibility, monitor warning signs, and prepare coordinated responses to continued AI progress.

---

How are AI capabilities progressing?

AI capabilities have grown rapidly in recent years. Experts disagree on whether this pace will continue, with some expecting even faster progress. Still, any future-proof plan needs to seriously account for the possibility that AI capabilities will continue to increase very rapidly. If they do, we are facing massive societal transformations on the order of a compressed Industrial Revolution. 

Many early warning signs of AI’s societal impacts have already started to blink. Societal impacts can range from cybersecurity and biosecurity to information authenticity, persuasion, labor impacts, and loss of control. In some areas, like cybersecurity, we already have solid evidence of serious AI misuse risk, as the rate of discovered software vulnerabilities has gone up dramatically. 

In other areas such as the labor market, the observed societal impacts may still be ambiguous, but we can see clear early warning signs, such as AI models that can do a broad range of real knowledge economy tasks as well as human experts. 

Finally, we see a clear trend towards long-range autonomy. The recent autonomous hack of HuggingFace by AI models from OpenAI shows that it’s worth taking the prospect of loss of control over AI models seriously.

Corporate responsibility from AI model producers is necessary but insufficient

Producers of frontier AI models have published plans with mitigating measures that increase as AI capabilities increase. Anthropic calls this its Responsible Scaling Policy, OpenAI calls this its Preparedness Framework, and international AI summits have referred to these as Frontier AI Safety Frameworks. 

These frameworks are important, but arguably corporate social responsibility of AI model providers is not sufficient, for various reasons including:

• Distributed societal impacts: Many potential AI challenges arise not from one AI model and one user, but from the cumulative use of many AI systems across society. For example, labor market displacement or correlated financial risks from AI investment advice are challenges that grow with adoption of AI models at scale.
• Irreversible proliferation: For threats requiring only a specific absolute capability, not the relative superiority of frontier capabilities, economic competition and the technological feasibility of exponential cost decay of intelligence favor open diffusion regardless of what any single actor does. This means we should expect increasing absolute AI capabilities that no company can easily roll back.
• Other control points for risks exist: Certain risks are more effectively managed beyond the AI model itself. For example, AI foundation models should by default not be able to generate child sexual abuse material. However, because threat actors can remove safeguards from open-weight models and fine-tune them for malicious capabilities, this is not sufficient to prevent the creation of AI CSAM. We must also rely on AI model hubs to remove re-uploaded AI models that were designed for that purpose, and on social media platforms and Internet Service Providers to deny access to such content.

These reasons do not absolve AI producers of their responsibility, but highlight that the most effective risk management will be multi-layered and involve a range of actors. Governments are and will remain one of these actors.

The case for a scenario-based, governmental AI preparedness process

Legislatures can regulate AI both as a general-purpose technology (e.g., EU AI Act Chapter V) and by domain application (e.g., FDA oversight of AI-enabled medical devices, EU AI Act Annex III). Similarly, national AI strategies developed by executive branches have often defined goals and actions for innovation and competitiveness (e.g., the EU’s AI Continent Action Plan, America’s AI Action Plan). 

However, neither is equivalent to the anticipatory and conditional logic of Preparedness Frameworks for advanced AI developed by AI producers: For example, if AI capabilities reach the point where they can significantly help individuals or groups with basic technical backgrounds create CBRN weapons, then companies plan to introduce stronger cybersecurity protections. Governments – whether in legislative or executive branches –  still lack a systematic, cross-cutting, and forward-looking approach to anticipate and manage societal impacts of AI. 

Governments have existing scenario-based playbooks for topics such as large-scale cybersecurity incidents or pandemic preparedness, but these plans do not account for the prospect of transformative AI and should be updated accordingly. Furthermore, advanced AI also comes with novel hazards or issues without clear ownership, such as preparing for loss-of-control scenarios. 

Given that most governments have limited in-house AI capacity, it is arguably better to have a concentrated effort (e.g., UK AI Scenarios 2030) to think through scenarios and response options (e.g., How Should the US Prepare for Increasingly Automated AI R&D?). Such a process could also force actors relevant to different aspects of AI to work more closely together. 

As with other governmental playbooks, reality will most likely not follow the exact scenarios outlined in a responsible scaling playbook for governments. However, it is still worth identifying societal challenges and preparedness measures to improve coordination between stakeholders, to identify gaps that need to be closed, and to understand a range of mitigation options. 

Governments should therefore consider a dedicated scenario process that assigns ownership for currently unowned risks and defines in advance which societal indicators trigger which responses. 

The first concrete steps to launch such a process are: 

1. Designating a lead coordination body: A specific body needs to lead the coordination between ministries. That could be, for example, a new dedicated unit, a national security body, the Prime Minister’s office, or an existing AI policy body with an expanded mandate.
2. Appointing a responsible individual: Regardless of the institutional home, an empowered individual needs to own this function. A “National AI Preparedness Coordinator” or equivalent needs sufficient understanding of AI challenges and trajectories, direct access to the head of government, authority to convene cross-departmental working groups, budget for preparedness activities, and the responsibility for regular reporting to the legislature.
3. Establishing a cross-government working group: A regular coordination mechanism bringing together relevant ministries and agencies, including National security, national AI testing capacity, economy & finance, public health, foreign affairs, and critical infrastructure regulators. The working group should be able to convene private sector and academic expertise as needed (e.g., frontier AI developers, cloud providers, and major ISPs).

Establishing these steps at the domestic level across governments also provides the foundation for governments to interface with each other through institutions and individuals with corresponding levels of authority, enabling transnational risk mitigation.