A company’s team of researchers has discovered what they say is the first reported use of artificial intelligence (AI) to direct a hacking campaign in a largely automated manner.
Artificial intelligence company Anthropic reported this week that it disrupted a cyber operation that its researchers linked to the Chinese government. The operation involved the use of an AI system to direct hacking campaigns, which researchers called a disturbing development that could greatly expand the reach of AI-equipped hackers.
While concerns about using AI to power cyber operations are not new, what is concerning about the new operation is the degree to which AI was able to automate some of the work, the researchers said.
“While we predicted that these capabilities would continue to evolve, what has surprised us is how quickly they have done so at scale,” they wrote in their report.
The operation targeted technology companies, financial institutions, chemical companies and government agencies. The researchers wrote that the hackers attacked “approximately 30 global targets and were successful in a small number of cases.” Anthropic detected the operation in September and took steps to shut it down and notify affected parties.
Anthropic noted that while AI systems are increasingly being used in a variety of work and leisure environments, they can also be weaponized by hacking groups working for foreign adversaries. Anthropic, creator of chatbot of Generative AI Claude, is one of many technology companies promoting AI “agents” that go beyond a chatbot’s ability to access computing tools and take actions on a person’s behalf.
“Agents are valuable for everyday work and productivity, but in the wrong hands, they can substantially increase the viability of large-scale cyberattacks,” the researchers concluded. “These attacks are only likely to grow in effectiveness.”
A spokesperson for China’s embassy in Washington did not comment on the report.
Microsoft warned earlier this year that foreign adversaries were increasingly adopting AI to make their cyber campaigns more efficient and less labor-intensive. The head of the security panel OpenAIwhich has the authority to stop AI development from the creator of ChatGPTrecently told The Associated Press that he is on the lookout for new AI systems that give malicious hackers “much higher capabilities.”
America’s adversaries, as well as criminal gangs and hacking companies, have exploited the potential of AI, using it to automate and enhance cyberattacks, to spread inflammatory disinformation, and to penetrate sensitive systems. AI can translate poorly worded phishing emails into fluent English, for example, as well as generate digital clones of top government officials.
Anthropic said the hackers were able to manipulate Claude, using “jailbreaking” techniques that involve tricking an AI system into bypassing its barriers against harmful behavior, in this case by claiming they were employees of a legitimate cybersecurity firm.
“This points out a big challenge with AI models, and it’s not limited to Claude, which is that models need to be able to distinguish between what’s really happening with the ethics of a situation and the types of role-playing scenarios that hackers and others may want to invent,” said John Scott-Railton, senior researcher at Citizen Lab.
Using AI to automate or direct cyberattacks will also attract smaller hacking groups and lone hackers, who could use AI to expand the scale of their attacks, according to Adam Arellano, head of field technology at Harness, a technology company that uses AI to help clients automate software development.
“The speed and automation provided by AI is what’s a little scary,” Arellano said. “Instead of a human with well-honed skills trying to hack hardened systems, AI is speeding up those processes and overcoming obstacles more consistently.”
1/12 | How to create an image using Microsoft artificial intelligence: Copilot. Among the capabilities of Microsoft’s artificial intelligence, Copilot, is the ability to generate images like this according to the instructions and needs of the users. – artificial intelligence
Artificial intelligence programs will also play an increasingly important role in defending against these types of attacks.Arellano said, demonstrating how AI and the automation it enables will benefit both sides.
Reaction to the company’s report was mixed, with some seeing it as a marketing ploy for Anthropic’s focus on cybersecurity defense and others welcoming its wake-up call.
“This is going to destroy us, sooner than we think, if we don’t make AI regulation a national priority tomorrow,” Sen. Chris Murphy, D-Conn., wrote on social media.
That led to criticism from Meta’s chief AI scientist, Yann LeCun, an advocate of the parent company’s open source AI systems. Facebook which, unlike Anthropic’s, make their key components publicly accessible in ways that some AI security advocates consider too risky.
“You are being manipulated by people who want to capture regulation,” LeCun wrote in response to Murphy. “They are scaring everyone with dubious studies so that open source models are regulated until they disappear