Policy Gradients Outperform Game Theory in Complex Imperfect-Information AI

New research co-authored by MIT and presented at the International Conference on Learning Representations (ICLR) in April reveals that neural networks trained with policy gradient algorithms can achieve superior performance in complex imperfect-information, zero-sum games compared to those leveraging traditional game theory-based approaches. This work offers significant insights into how AI agents can navigate strategic environments where complete information is unavailable.

Advancing AI in Strategic Games

The research, spearheaded by MIT's Sobhan Mohammadpour and Gabriele Farina, alongside collaborators from the University of Texas at Austin, University of California at Berkeley, Carnegie Mellon University, and New York University, addresses a long-standing challenge in AI: enabling agents to make optimal decisions in games with hidden information. Unlike perfect-information games where all players know the full state, imperfect-information scenarios, akin to poker or espionage, require agents to infer opponent strategies and manage uncertainty effectively. The study specifically focuses on training neural networks to excel in these intricate competitive settings.

Novel Benchmarking and Surprising Results

A pivotal contribution of this research is the development of a novel benchmark designed to rigorously assess various algorithms for teaching neural network agents how to compete. This benchmark quantifies an agent's prowess using 'exploitability,' a metric that measures how effectively a player performs against a theoretical worst-case adversary. A perfect score of zero exploitability signifies flawless play. The team applied this benchmark to five distinct imperfect-information games, including two versions of Phantom Tic-Tac-Toe, two variants of Hex, and Liar's Dice. Notably, some of these test cases involved game states numbering up to 30 billion, a scale vastly exceeding the typical scope for exploitability analysis in prior research by factors of 100,000.

The experimental findings were compelling. Neural networks trained using policy gradient methods, which have been applied to decision-making problems since the 1990s, consistently achieved lower (better) exploitability scores. Furthermore, in direct head-to-head competitions, these policy gradient-trained networks unequivocally defeated their counterparts that relied on game theory-based algorithms.

Accessible Tools for Researchers

To facilitate broader research and development in this domain, the team has made their benchmarking software openly available. It is designed for ease of use, capable of running on standard laptops, and seamlessly integrates with OpenSpiel, a widely adopted collection of benchmarking software for game AI, requiring only a single line of code for implementation. This accessibility stands to accelerate further exploration into the efficacy of different training paradigms for AI agents in complex strategic environments.