Abstract
Multimodal video-audio-text understanding and generation can benefit from datasets that are narrow but rich. We present MUGEN, a large-scale dataset collected using a modified version of the platform game CoinRun. MUGEN consists of 375,000 video clips (3.2s each) with synchronized audio, pixel-perfect semantic maps, and both manual and automated text descriptions. By training 14 reinforcement learning (RL) agents with diverse objectives, we ensure a wide variety of interactions and behaviors. We benchmark representative approaches on retrieval and generation tasks, providing a foundation for future research in spatiotemporal multimodal reasoning.
Introduction
Progress in multimodal AI has seen significant breakthroughs in image-text domains, driven by massive datasets and models like CLIP and DALL-E. However, video remains a challenging frontier. Real-world video datasets are often noisy, with misaligned audio or subjective text descriptions. Conversely, simple synthetic datasets like Bouncing MNIST lack the complexity required for sophisticated reasoning.
MUGEN addresses this gap by providing a "Goldilocks" environment: controlled enough for objective evaluation, yet rich enough to capture complex interactions between multiple entities and modalities. It bridges the gap between toy synthetic data and the chaotic "in-the-wild" web videos.
The MUGEN Dataset
MUGEN is built upon a heavily modified version of CoinRun. To generate diverse data, the researchers trained 14 RL agents with distinct reward functions—ranging from "risk-tolerant" agents that frequently die to "myopic" agents with limited foresight. This approach produced a vast array of gameplay scenarios across procedurally generated levels.
Each 3.2-second clip in the dataset is a multimodal powerhouse. It includes RGB video frames, synchronized audio triggered by game events (e.g., jumping, collecting coins), and pixel-perfect semantic maps. Textual descriptions are provided in two forms: "manual text" written by human annotators and "auto-text" generated from game engine metadata, ensuring both natural language variety and ground-truth precision.
Dataset Diversity and Statistics
The primary contribution of MUGEN is its scale and richness. The dataset includes 375K video clips and 379K manual text descriptions. Because it is built on a game engine, the ground truth is absolute; the system knows exactly when a character jumps or interacts with an object. This allows for precise evaluation of model faithfulness.
The dataset features 13 different objects and characters, including coins, gems, and various monsters like snails and bees. The distribution of actions (jump, collect, kill, etc.) is carefully tracked to ensure the dataset supports complex spatiotemporal reasoning tasks.
Benchmarking and Results
The authors evaluated MUGEN using a VQ-VAE to tokenize video and audio into discrete representations, followed by an autoregressive transformer to predict tokens across modalities.
- Retrieval: Video-to-audio and audio-to-video retrieval performed exceptionally well (R@1 ~60%), likely due to the tight synchronization of game events. Text-to-audio proved to be the most challenging retrieval task.
- Generation: In Text-to-Video (T2V) tasks, human evaluators chose MUGEN-generated videos over ground truth 31.25% of the time for quality and 42.6% for faithfulness.
- Efficiency: Because the data can be stored as game metadata (JSON) rather than raw video, it is over 100x smaller in storage size than lossless high-resolution video, making it highly accessible for researchers.
A major insight from the experiments was that models often generate "faithful" content (matching the text prompt) even if the visual reconstruction quality is limited by the VQ-VAE bottleneck. This suggests that future improvements in generative architectures, such as Diffusion Models, could further enhance the visual fidelity of MUGEN-based generations.
Conclusion
MUGEN provides a scalable, controllable, and rich playground for the next generation of multimodal models. By bridging the gap between toy datasets and the chaotic real world, it enables researchers to focus on the core challenges of synchronized spatiotemporal generation and understanding. The authors have released the modified game engine and dataset to encourage community expansion into new worlds and interaction types.
Summary updated