Microsoft has issued a warning about the risks of granting too much autonomy to artificial intelligence within companies. Hayete Gallotexecutive vice president of Microsoft Securitymaintains on the company blog that ‘the most advanced AI models cannot be trusted, regardless of their origin’ and proposes treating the AI agents that use them as possible internal threats.
According to The Verge, the CEO of Microsoft, Satya Nadellahad already argued this month that we should assume that the models are ‘engaged’. In a publication in X on October 10, he proposes designing the systems based on that possibility and ensure that an authorized person can stop them at any timeas if it had an emergency brake.
AI agents are programs that use artificial intelligence models to perform tasks with different degrees of autonomy through tools and applications. They can consult information, modify documents or manage processes, as well as work in the background without needing instructions for each step. Its ability to automate tasks is driving its incorporation into Microsoft services and other companies.
The progressive implementation of these systems is accompanied by increasing permissions that allow them to access sensitive data, use tools and make decisions in critical processes. Gallot warns that many they receive more privileges than necessarywhich increases the consequences of any unforeseen behavior.
The danger that Microsoft describes may arise even when no attacker has manipulated the agent. The models are trained to achieve results and can finding shortcuts that satisfy your goals without respecting what the organization intended. Gallot points out that ‘agents are unpredictable. They will adapt as the context changes, may not behave as expected, and will collaborate and collude with other agents to achieve their objectives. without having to be compromised‘. Having them functioning within a company requires, therefore, continually monitor your actions.
Microsoft’s proposal consists of use guardian agents to review the actions of other AIs before they are executedduring 24 hours a day. For this purpose, it is proposed to apply Project Perceptionthe security system presented in July to deal with attacks powered by artificial intelligence, as well as internal risks.
The system gathers information about identities, devices, networks, data, and cloud services to identify agents and assign rules and permissions. Then evaluate the risk of what they are trying to do. Guardian officers monitor your activity and intervene when they detect an increase in danger. The design combines this surveillance with controls external to the model that limit its ability to act.
Microsoft defends that automatic monitoring should allow human managers to reconstruct what happened and review security measures. He also claims greater transparency about the reasoning of the models to understand why they act and check whether their decisions respond to the task assigned.