When AI Agents Start Interacting at Scale: Implications for AI Governance 

This post is the first of a series on the international governance of frontier AI by our CEO, Max Stauffer. In this series, Max combines his experience advising governments on frontier AI diplomacy and his background in understanding complex, multi-agent systems.

---

In this blog post, I describe what I see as a logical sequence of events in which AI agents from different companies and countries start interacting. I frame this future trajectory as a necessary area of frontier AI diplomacy that countries or companies are unlikely to address by implementing parallel measures alone.  

The AI incidents of the past months – and in particular the OpenAI/Hugging Face incident – confront AI policy communities with a well-known governance puzzle. Even though the type of those incidents was predicted, policy action has been lacking. We’re now seeing a spike in policy interest because incidents have grown more salient, but this reactive dynamic only works if incidents aren’t overly severe or irreversible. 

Given the social and bureaucratic fabric of policy, governments are unlikely to escape their reactive mode anytime soon, exposing themselves to potentially irreversible shocks. The solution isn’t to abandon policy. Instead, governments can change how they react by learning what observed incidents teach us about what will happen in the future. To exemplify, I first briefly unpack the OpenAI/Hugging Face incident, then extrapolate a logical sequence of future events, and finish with recommendations for policymakers with particular attention to frontier AI governance. 

A High-Level Anatomy of the OpenAI/Hugging Face Incident

Between May and July 2026, groups of AI agents inside OpenAI self-organized, in part to cheat on their safety benchmarking tests. The incident – as we currently know it – unfolded in three phases. 

Phase 1: In May, copies of an AI model being trained for persistence learned to interact and talk to each other via a shared software tool that served as a communication channel, exploiting their interactions to reach the internet. Model training inadvertently rewarded those persistent behaviors. 

Phase 2: In July, AI agents were launched on a cybersecurity benchmark (ExploitGym) with often impossible tasks. Stuck agents rebuilt their hidden communication channel, at a scale of 1,200 interacting agents exchanging 70,000+ messages, and managed to regain internet access. They then spent days coordinating to falsify their logs to hide any evidence of cheating. One branch of that became an autonomous, end-to-end intrusion into Hugging Face (a collaborative hub and open-source platform for AI research). As far as we know, this is the first publicly verified case of AI systems targeting an uninvolved third party and hacking it without human direction. 

Phase 3: Later in July, more capable groups of AI agents rediscovered the abandoned communication channel and leveraged it to hack OpenAI itself, gaining administrator access to an internal research cluster. 

Overall, this happened because a partially trained behavior enabled coordination and kept agents from alerting humans to their covert, possibly illegal, actions. Additional sources are listed at the end of this post. 

A Rapid Review of AI Developments for Policy Audiences

Before delving into what this incident teaches about the future, it is important to recap how AI systems became capable of organizing themselves as swarms of agents. Even actors closest to the source who have researched safety failures for years are still trying to understand what exactly happened. This makes the nature and significance of the OpenAI/Hugging Face incident complicated and technical to understand. 

The emergence of swarms of agents results from at least four key developments; if this is clear to you, feel free to skip to the next section.

1. Chatbots based on large-language models (LLMs): The AI systems behind chatbots are trained to predict the next word across enormous amounts of text, then tuned with human feedback to be helpful and safer. This development led to the release of ChatGPT in 2022 and Claude and DeepSeek V2 in 2023. Chatbots took questions, provided answers, and interactions remained between AI systems and humans. Developers refined the quantity and quality of those answers by training models at a bigger scale using more compute and more data, leading AI models to match or exceed expert performance on many benchmark tests.
   
2. Reasoning models: In 2024, AI developers began training models not just to sound right to a human and give correct answers, but to complete long, multi-step tasks, which is the approach behind “reasoning” models. Performance increased rapidly, with METR reporting that the length of tasks an AI can complete on its own has been doubling roughly every seven months. This development made autonomy viable, because long chains of reasoning or actions only succeed if each step is reliable enough. By now, it has become difficult to know how well models perform because benchmarks have saturated and likely fail to test for all their capabilities.
   
3. Agents: Coding was the main domain where AI developers applied the above approach. Coding is an ideal training ground because it is multi-step and easily testable through many verifiable test cases. Additionally, code is a universal lever in the digital world: almost anything can be operated by writing and running a script. Therefore, an AI model that codes reliably can do far more than code. From there, AI models went from saying to doing things, thus becoming AI agents.
   
4. Multi-agent systems: Lastly, AI developers then enabled individual agents to interact. First, they gave AI models a way to call external tools and, with the Model Context Protocol released in late 2024, enabled this functionality without custom integration for every tool. Second, developers discovered that many specialized agents working in parallel can outperform one generalist, enabling agents to call not only tools but also other agents.

For more information on AI trajectories, the underlying and resulting trends, check Epoch AI. For information on risks specifically related to multi-agent AI systems, see this report. 

Capabilities and Scale Will Shape Future Incidents

The OpenAI/Hugging Face incident happened almost entirely inside one company. Every agent involved was a copy of one developer’s model, running on that developer’s infrastructure, in one country. And yet, this contained case was barely legible. OpenAI, with full access to its own logs, needed weeks and external help to understand what its own agents had done. This situation is the starting point. Moving forward, future incidents will depend on at least two dimensions: the development of AI capabilities and the scale at which they’re deployed. 

AI capabilities development

• Questions: What will new models be able to achieve? How persistent will they be? Can developers ensure safety at the individual-agent and multi-agent levels? How can we harmonize safety measures across companies and countries? 
• Current state: We know AI companies have sufficient compute, data, and talent to train more powerful models, possibly even automating AI R&D itself. By default, we should expect greater model capabilities to cause incidents more severe than the OpenAI/Hugging Face incident unless significant progress is made in agent and multi-agent safety. 
  

Scale of AI deployment

• Questions: Will companies continue to deploy ecosystems of agents internally and let them interact? What happens when hundreds, thousands, or millions of agents interact? Will the scale of agent deployment feed back into agent-level capabilities? Are there scales beyond which human control ends, and is it possible to know those scales in advance? And will those multi-agent systems scale within companies only, or span multiple companies and possibly countries? 
• Current state: Most incentives in most countries point to the large-scale adoption of AI systems across sectors. By default, we should expect automation progress to lead to multi-agent ecosystems that cross industry and national borders (more below).

If the capabilities and deployment scale of AI agents continue to grow, the consequences will extend far beyond individual companies. These systems could create immense value, improving lives through better coordination at scale. But transformative AI also introduces the possibility of irreversible harm. Cybersecurity offers an early view of how these risks can compound: advanced models can accelerate the discovery and weaponization of software vulnerabilities (see recent example), while automated attacks can spread through trusted networks faster than human defenders can respond. As agents gain access to digital systems and critical infrastructure, the boundary between a software failure and a societal crisis narrows. The central risk is loss of control: a point at which humans can no longer govern agents capable of deploying or leveraging other highly capable agents, operating across institutions, and accessing the critical infrastructure modern societies depend on.

When AI Agents Interact Between Countries    

AI capabilities and the scale of agent deployment will likely progress hand in hand. Many have tried to forecast the implications of further progress in AI capabilities, in particular when AI capabilities lead to the automation of AI R&D itself. As a complement, below I focus on what I see as the logical sequence of events primarily related to the scale of AI deployment. The OpenAI/Hugging Face incident took place among multiple AI agents within one company and one country. I believe that, by default, we should expect that agents will span different companies and especially countries. Consider the following progression: 

1. Within one company: Currently, groups of agents primarily interact within individual companies, where a single actor holds all logs (as in the OpenAI/Hugging Face incident). Sense-making depends on internal teams plus invited independent investigators. Measures include better safety practices, ideally with each company securing itself. Third-party actors and governments help ensure standardization and accountability.

2. Across companies: As agents become each other’s tools – a coding agent from one firm calling a service maintained by another firm’s agent, a procurement agent negotiating with a sales agent – incidents will emerge increasingly from interactions between systems built and operated by different organizations. At that point, no single actor would hold the full picture. Deciphering an incident requires piecing together logs held by different companies, likely in different formats, under different security and legal frameworks, and with different incentives to cooperate. Sense-making, which is already hard inside one company, fragments across several.
   
3. Across countries: From there, multi-agent ecosystems across countries seem very likely because agents are being deployed into an economy already built on cross-border interactions and increasingly run on software. Current trade infrastructures – supply chains, payments and settlements, insurance, cloud services, enterprise software, and more – involve firms in multiple jurisdictions that exchange messages many times each day. On top of that, countries are seeking to leverage AI for their economies, even if they are not a frontier developer. Shared infrastructure like the Model Context Protocol makes agent-to-agent interactions more likely and faster. Finally, the technical infrastructure for research is global, via code repositories, package registries, and model hubs that agents can read from and write to, enabling them to interact.

The same interaction logic applies between the US and China. The natural objection is that the two ecosystems are divided because of separate infrastructure and geopolitical tensions. US-China AI integration remains limited: major US AI services are not officially available in mainland China, while Chinese AI services remain accessible to US consumers but face restrictions in government and other sensitive settings. This separation does not, however, affect the infrastructure on which agents would meet. Hundreds of billions of dollars in goods are traded each year between the two countries, with much of that trade running on software. Firms on both sides communicate by email over the open internet, and their developers depend on the same code repositories, package registries, and model hubs. Additionally, offensive and defensive cyber agents will encounter each other regardless of trade policy.

Agents interacting across countries will raise the question of jurisdiction. In case of an incident, the sense-making questions become international and possibly geopolitical. Whose agents started it? Was this a misaligned system pursuing a mistaken goal, as at Hugging Face, or a deliberate operation? Again, no single company holds the transcripts, and no single government holds the facts. Attribution becomes a matter on which capitals may disagree, creating surface area for false accusations; time pressure will conflict with the slow pace of international cooperation, and success will depend heavily on trust. Therefore, we will ultimately end up with an international economy of deployed AI agents whose interactions produce potential incidents. This structure defines the form of AI diplomacy needed to solve it.

Multi-Agent Risks as a Necessary Area of Frontier Diplomacy

Most of what we ask of AI governance today can be done in parallel. Each country can require its developers to test models before release, set a safety floor, and hold companies accountable before and after something goes wrong. Two governments that do not trust each other can each do this work, compare notes where possible, and both come out better. While this agenda could work for now, it is far from complete. Once agents from different countries interact and generate risk, no country can assess and mitigate that risk alone, not because of a lack of goodwill or trust, but because the relevant facts and points of leverage are physically dispersed. A government that perfectly regulates every developer within its borders will, on its own, be unable to tell whether an incident originated from a misaligned agent or a deliberate operation. That is a problem no amount of domestic diligence solves.

Parallel governance arrangements such as the 1994 Convention on Nuclear Safety have clear limitations: states report on their own reactors for peer review, with no inspections, because governments would not accept joint authority over domestic assets. The AI agent deployment progression above requires a response to risks that arise between systems, since no single state has full control. We can learn from existing measures and initiatives for mitigating transnational risks. Examples include the Comprehensive Nuclear-Test-Ban Treaty (CTBT) monitoring network, the International Civil Aviation Organization (ICAO) accident-investigation framework, and the International Health Regulations. One similarity is that they involve building a shared understanding of risks, risk reporting and disambiguation, and potential measures to implement in a harmonized fashion. 

Below are three recommendations that countries can take forward to address the progression to multi-agent ecosystems across countries:

1. Investing in agent monitoring: To better govern AI systems, countries need to better understand deployment scales. Currently, no government knows how many agents are deployed in its own jurisdiction, what they do, or how they interact. Building the national capacity to observe deployment at the system level is the precondition for everything else: monitoring, reporting, and joint investigation. Internationally, the requirement is not necessarily a single global body but interoperability so countries’ observations can be compared.
   
2. Establishing agreed channels for cross-border incidents: Once incidents span jurisdictions, the question is not only what happened but how states make sense of, and communicate about it. States could agree on formats and channels for reporting cross-border agent incidents to disambiguate them. Further, states could create an accepted procedure for determining whether an incident was a misaligned system pursuing a mistaken goal or a deliberate operation. The purpose is defensive: to prevent a technical accident from being read as an attack, and to keep an ambiguous event from escalating faster than either government can assess it.
   
3. Investing in incident prevention and AI assurance: As deployment scales and capabilities advance, the joint agenda extends to reducing the probability of incidents in the first place. States should invest in convergent safety practices for multi-agent deployment, shared technical standards for agent identity, logging, and agreed thresholds beyond which additional precautions apply.

For more guidance on multi-agent AI governance, I recommend this report from Australia’s AI Safety Institute.

The Road Ahead

The reactive pattern of governance, waiting for incidents, then responding, is unlikely to change. But it is possible to change the quality and focus of each reaction. The OpenAI/Hugging Face incident was contained, investigated, and voluntarily disclosed; it is a comparatively gentle case. More incidents will likely happen, probably with a higher severity. Therefore, to learn from this Summer’s incident, we shouldn’t only fix what went wrong at one company, but read forward from what it shows about the structure of the problem in the future, internationally. That lens clarifies why reducing multi-agent AI risks is a necessary area of frontier AI diplomacy.

OpenAI/Hugging Face Incident Primary and Secondary Investigations and Analysis 

• Developer’s perspective: OpenAI’s technical report and summary post 
• Independent, third-party analysis: METR and Redwood Research investigation (specifically the Hugging Face attack) 
• Hugging Face recaps the intrusion from the target’s side. 
• Dwarkesh Patel’s walkthrough and his interview with METR’s Ajeya Cotra (as well as her own reflections), and Ezra Klein’s podcast with Helen Toner
• Institute for AI Policy and Strategy on policy implications
• ChinaTalk on China’s reaction (specifically concerning the Hugging Face incident)