Abstract
Developing spatial reasoning capabilities in multi-agent environments is a significant challenge for AI research. While simple datasets like MNIST lack strategic depth, high-fidelity simulators are often computationally prohibitive for rapid prototyping. We present StarCraftImage, a massive dataset derived from 60,000 human-played StarCraft II replays. By summarizing 3.6 million game windows into hyperspectral, RGB, and grayscale image formats, StarCraftImage provides a standardized, "plug-and-play" platform for spatial reasoning. The dataset includes a "bag-of-units" representation to handle dense spatial data and built-in corruption models to simulate real-world sensor noise. Our results demonstrate that StarCraftImage serves as an effective proxy for real-world tasks, such as satellite imagery analysis.
Introduction
Multi-agent systems, such as autonomous vehicle fleets or drone swarms, require sophisticated spatial reasoning: the ability to identify agent types, track positions, and predict movement patterns amidst environmental obstacles. Current research is often bifurcated between overly simplistic toy datasets and resource-intensive simulators. StarCraftImage bridges this gap by distilling the adversarial complexity of a professional real-time strategy game into accessible image formats. Unlike synthetic simulations, these replays capture high-level human strategic intent and complex micro-management.
Methods and Data Representation
The dataset construction focuses on summarizing temporal sequences into static spatial representations.
Temporal Summarization
Instead of processing raw video, the authors summarized windows of approximately 10 seconds (255 game states) into single "summary images." A "last seen" timestamp is recorded for each unit at each coordinate. This creates a "ghosting" effect that implicitly encodes movement and velocity within a single 2D frame.
Logical View and Bag-of-Units
To manage the high dimensionality of StarCraft II (hundreds of unit types), the data is organized into a logical view separating players and neutral units. A "bag-of-units" representation—analogous to bag-of-words in natural language processing—is employed to handle instances where multiple units occupy the same spatial coordinate, ensuring no data is lost in dense clusters.
Multi-Format Support
To cater to different research needs, the dataset is provided in three primary formats:
- StarCraftHyper: A 384-channel hyperspectral format containing detailed unit and terrain information.
- StarCraftCIFAR10: A 32x32 RGB version where colors represent players (Blue=P1, Red=P2, Green=Neutral).
- StarCraftMNIST: A 28x28 grayscale version using pixel intensity for unit and player differentiation.
Experimental Results
Baseline benchmarks using ResNet and U-Net architectures were established for tasks such as unit identification and game phase classification.
- Performance: Models achieved ~56-62% accuracy on clean unit identification tasks. For 10-way classification (map and game phase), simple ConvNets reached ~78% accuracy.
- Robustness: The inclusion of corruption models (noise, masking, blur) revealed significant vulnerabilities. Accuracy dropped to ~30% under structured partial observability (e.g., diagonal barriers), indicating that current models struggle with complex spatial occlusions.
- Real-World Transferability: A key validation of the dataset is its transferability. Models trained on StarCraftImage were successfully applied to the DOTA-UnitID satellite dataset. The consistency in model rankings between these two datasets suggests that StarCraftImage is a valid proxy for real-world aerial spatial reasoning.
Conclusion and Future Work
StarCraftImage provides a scalable middle ground for multi-agent AI research. While the collapse of 10-second windows into static images loses some fine-grained temporal dynamics, the "last seen" encoding and bag-of-units approach preserve essential strategic information. Future work will explore expanding the dataset to full video sequences and developing adversarial attacks tailored for multi-agent spatial contexts.
Dataset and code available at: https://starcraftdata.davidinouye.com/
Summary updated