Google's latest AI plays video games alongside you, but not competitively

Google DeepMind trained its AI gaming agent using titles such as Valheim, No Man’s Sky, and Goat Simulator.

Google DeepMind has introduced SIMA, an AI agent designed to learn gaming skills to emulate human-like gameplay rather than exhibiting overpowering AI behavior. SIMA, an acronym for Scalable, Instructable, Multiworld Agent, is currently in the research phase.

SIMA aims to eventually master playing any video game, including those without a linear progression or featuring open-world environments. While not intended to supplant existing game AI, SIMA is envisioned as an additional player that seamlessly integrates into gaming experiences. It combines natural language instructions with the ability to comprehend 3D environments and recognize images.

According to Tim Harley, a researcher at Google DeepMind and co-leader of the SIMA project, "SIMA isn't trained to win a game; it's trained to run it and follow instructions."

Google collaborated with eight game developers, including Hello Games, Embracer, Tuxedo Labs, and Coffee Stain, among others, to train and evaluate SIMA. The AI agent underwent training in games like No Man’s Sky, Teardown, Valheim, and Goat Simulator 3 to grasp fundamental gameplay mechanics. Google emphasized in a blog post that SIMA doesn't require a custom API or access to source codes to play these games. 

Harley explained that the team selected games emphasizing open-ended gameplay rather than narrative-driven experiences to assist SIMA in acquiring general gaming proficiency. For instance, Goat Simulator, known for its spontaneous and random gameplay, was chosen to instill such spontaneity in SIMA.

To achieve this, the team constructed a new environment in the Unity engine, where agents were tasked with creating sculptures to evaluate their grasp of object manipulation. Subsequently, Google recorded pairs of human players — one controlling the game and the other issuing instructions on the next actions — to capture verbal instructions. Following this, players engaged in independent gameplay to elucidate the rationale behind their actions in the game. All this data was then utilized to train SIMA agents to predict forthcoming events on the screen.

Currently, SIMA possesses around 600 fundamental skills, including actions like turning left, climbing ladders, and accessing menus to use maps. Harley indicated that in the future, SIMA could be instructed to perform more intricate functions within a game. However, tasks such as "finding resources and constructing a camp" remain challenging because AI agents cannot execute actions on behalf of humans.

SIMA is not designed to function as an AI-powered non-player character (NPC) akin to those developed by Nvidia and Convai; rather, it operates as another participant in the game, influencing the outcome. Frederic Besse, co-lead of the SIMA project, stated that it is premature to predict the potential applications of AI agents like SIMA beyond the realm of research.

Although akin to AI NPCs, SIMA may eventually acquire conversational abilities; however, it is currently in the early stages of learning to play games and adapting to new ones. Google mentioned that with more advanced AI models, SIMA could potentially perform more complex tasks and serve as an adept AI companion to guide players to victory.