It’s the kind of event once only seen in science fiction: an artificial intelligence system, trained to detect digital vulnerabilities, breaks free from human control and acts on its own to hack another company.
The attack announced this week by OpenAI, which blamed rogue AI models, highlighted the rapid growth of the technology’s capabilities. For many, it also added urgency to questions about whether and how to prevent it from causing chaos on a larger scale, with more serious consequences.
In what OpenAI called an “unprecedented” episode, the company claimed that its advanced AI models used stolen credentials to access the servers of an AI startup. It all started in what was supposed to be a “very isolated” testing environment, with reduced security measures, before the AI agent managed to access the Internet.
However, this revelation was an “I told you so” moment for researchers who have called for a halt to the development of AI and have been warning for years that this technology could pose existential risks to humanity. As a result, experts have called for AI companies to improve their testing and for greater dialogue between the United States and China to find common solutions.
“I think we should take this as a warning sign not to make them smarter, and that will probably require global collaboration,” said Nate Soares, co-author of the 2025 book titled “If Someone Builds It, Everyone Will Die.”
Cyberattack could force companies to improve their containment measures
If a model can decide on its own to do something unethical, illegal, or harmful, what, if anything, can humans do to stop it from doing so?
OpenAI said it had ordered the AI models involved to carry out “advanced exploitation using complex attack paths” to test their cyber capabilities, but the technology went far beyond what was anticipated. He apparently decided on his own to attack Hugging Face, a well-known AI development center and marketplace, to obtain the information he needed to carry out a task.
Zahra Timsah, co-founder and CEO of governance platform i-GENTIC AI, said she hopes this incident will increase pressure on OpenAI and its competitors to conduct rigorous testing and study containment measures more thoroughly before AI systems are made available to the public.
Monitor agent behavior a posteriorias OpenAI is now doing with its research, is no longer enough, he said. “It’s like having the seat belt, the airbags, the brakes… everything that a car has. It should be there before the car starts driving,” Timsah said.
This revelation occurs in a context of growing concern about the cybersecurity capabilities of the most powerful models. In June, President Donald Trump signed an executive order establishing a framework for the federal government to evaluate the national security risks posed by the most advanced artificial intelligence systems for up to one month before their public launch.
Other experts consider that this event is an indication of the growth problems of AI
Some experts say the attack is part of the trial and error process involved in improving cybersecurity capabilities and is not cause for alarm.
“We’ve been dealing with people launching cybersecurity attacks for as long as the Internet has existed. And one of the interesting features of these language models is that the same capabilities that allow them to carry out cyber attacks also allow them to analyze cyber threats and create defenses against them,” said John Thickstun, an associate professor of computer science at Cornell University who studies methods to control the behavior of AI models.
This revelation has sparked skepticism among those who say it is in OpenAI’s interest to make its technology seem scarier. Given that OpenAI staff had decided to disable some security measures for the test, some have argued that the result should not have been a big surprise.
Thickstun noted that this disclosure responds to the need for OpenAI—a startup preparing to go public on Wall Street—to raise funds.
“The story they have been constantly telling throughout the history of this company is a story about how dangerous their models are, which their investors interpret as a story about how powerful their language models are,” he said.
This revelation reignites calls for greater regulation
The cyberattack has, in some circles, revived calls for greater regulation and oversight of artificial intelligence companies.
U.S. Rep. Greg Casar, D-Texas, wrote on social media: “We need mandatory, regular, independent security testing, adequate oversight, mandatory disclosure of security incidents, and international cooperation to protect people from unmitigated disaster.”
Soares, director of the Artificial Intelligence Research Institute, said that the United States will have to engage in conversations with its main AI competitor, China, something that, in his opinion, is not as far-fetched as it might have seemed just a year ago. Chinese leader Xi Jinping warned at a conference last week of the need to prevent AI from evading human control. And after an initial reluctance to regulate AI, the Trump administration has been more restrictive in containing cybersecurity risks.
“Many things can change when those responsible for national security begin to realize that they face a serious threat,” Soares said. “Will this make them realize it? Hopefully. I’m not sure. If this doesn’t do it, maybe the next incident will.”
AI pioneer Yoshua Bengio stated on social media that the episode is deeply worrying and should serve as a wake-up call.
“Continuing on the current trajectory of AI development will likely lead to an increase in specific cases of autonomous cyberattacks, as well as other high-risk incidents resulting from dangerous and uncontrolled AI behavior,” said Bengio, a professor at the University of Montreal. “We urgently need to take action to prevent these situations, rather than trying to repair the damage once it has occurred.”
This story was translated from English to Spanish with an artificial intelligence tool and was reviewed by an editor before publication.