A team of researchers from the Oxford University and other institutions, including Google DeepMind and the UK AI Safety Institutehas put four artificial intelligence models to play Civilization VI to see how they cope with tasks that require maintaining a strategy for hours. The balance was three victories and twenty defeats, with elementary failures for a human player such as not checking if another civilization was about to win. The study, called CivBenchwas published on arXiv on September 2, 2026.
The experiment used Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro and Kimi K2.5. Each model led a civilization against rivals controlled by the video game itself, without facing the other AIs in the experiment.
Civilization VI’s choice answers to the variety and quantity of decisions required during a game. This classic turn-based strategy saga allows you to develop cities, research technologies, negotiate agreements or wage wars. You can win through science, culture, religion or diplomacy, among other ways, without having to conquer all your adversaries.
The objective was to check How AIs tackle complex tasks that require chaining together many decisions and maintaining a plan for long periods of time. The models can explain game strategies, but the researchers wanted to find out if they were able to apply them and adapt them to what was happening. A game can exceed 300 turns and requires addressing various aspects at the same time, increasingly as it progresses.
Researchers developed a system that allowed AIs play using commands instead of the mouse. Models could request information about their cities and troops and indicate which unit to move or which building to build. The system executed those orders in Civilization VI and I returned the results in writing. AI models had 76 functions to perform specific actions, such as checking the location and status of your troops, moving a unit, founding a city, or choosing which technology to research. The system is available in the project repository.
The researchers gave all the AIs the same instructions, including checking every twenty turns to see if any rival was close to winning and recording their next steps in a diary that they could consult later. Liam Wilkinsonco-author of the study, explains that, in the first tests, The games accumulated so much information that the initial details were no longer available to the model.. That’s why they incorporated a journal that the AIs could consult to retrieve their plans. However, that help did not prevent them from leaving tasks they had planned undonesuch as building scientific facilities or founding another city.
One of the examples that relates Wilkinson shows an AI preparing a military strategy to Macedonia. For 110 turns he did not build the camp necessary to develop it, although he repeatedly acknowledged that he had to create the military infrastructure.. In another case, an AI insisted on its scientific superiority while The data showed that it was behind its rivals.
He threat tracking It also failed despite the researchers’ instructions. The models consulted how much players needed to win every 30 to 75 turnsalthough they had to do it every twenty. The information was available, but They had to request it to know it and they took a long time to do so..
The study points to a problem of how they distributed their attention. They focused on immediate actions and did not sufficiently review the overall situation or what they had decided to do. They do not identify a single cause nor do they demonstrate that better memory would be enough to solve it.
The authors consider that a AI with greater ability to solve problems could continue making these mistakes if you do not consult the necessary information or leave your plans pending. That is why they propose to try Automatic checks and to-do lists that stay up to dateso that the AI periodically checks its rivals and recovers tasks it has not yet completed.