MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration

Thomas Hayes, Songyang Zhang, Xi Yin, Guan Pang, Sasha Sheng, Harry Yang, Songwei Ge, Qiyuan Hu, Devi Parikh

Abstract

Multimodal video-audio-text understanding and generation can benefit from datasets that are narrow but rich. We present MUGEN, a large-scale dataset collected using a modified version of the platform game CoinRun. MUGEN consists of 375,000 video clips (3.2s each) with synchronized audio, pixel-perfect semantic maps, and both manual and automated text descriptions. By training 14 reinforcement learning (RL) agents with diverse objectives, we ensure a wide variety of interactions and behaviors. We benchmark representative approaches on retrieval and generation tasks, providing a foundation for future research in spatiotemporal multimodal reasoning.

Introduction

Progress in multimodal AI has seen significant breakthroughs in image-text domains, driven by massive datasets and models like CLIP and DALL-E. However, video remains a challenging frontier. Real-world video datasets are often noisy, with misaligned audio or subjective text descriptions. Conversely, simple synthetic datasets like Bouncing MNIST lack the complexity required for sophisticated reasoning.

MUGEN addresses this gap by providing a "Goldilocks" environment: controlled enough for objective evaluation, yet rich enough to capture complex interactions between multiple entities and modalities. It bridges the gap between toy synthetic data and the chaotic "in-the-wild" web videos.

The MUGEN Dataset

MUGEN is built upon a heavily modified version of CoinRun. To generate diverse data, the researchers trained 14 RL agents with distinct reward functions—ranging from "risk-tolerant" agents that frequently die to "myopic" agents with limited foresight. This approach produced a vast array of gameplay scenarios across procedurally generated levels.

Each 3.2-second clip in the dataset is a multimodal powerhouse. It includes RGB video frames, synchronized audio triggered by game events (e.g., jumping, collecting coins), and pixel-perfect semantic maps. Textual descriptions are provided in two forms: "manual text" written by human annotators and "auto-text" generated from game engine metadata, ensuring both natural language variety and ground-truth precision.

Overview of MUGEN's multimodal data components, including video, semantic maps, audio, and text.

Dataset Diversity and Statistics

The primary contribution of MUGEN is its scale and richness. The dataset includes 375K video clips and 379K manual text descriptions. Because it is built on a game engine, the ground truth is absolute; the system knows exactly when a character jumps or interacts with an object. This allows for precise evaluation of model faithfulness.

Distribution of characters, actions, and interactions within the MUGEN dataset.

The dataset features 13 different objects and characters, including coins, gems, and various monsters like snails and bees. The distribution of actions (jump, collect, kill, etc.) is carefully tracked to ensure the dataset supports complex spatiotemporal reasoning tasks.

Benchmarking and Results

The authors evaluated MUGEN using a VQ-VAE to tokenize video and audio into discrete representations, followed by an autoregressive transformer to predict tokens across modalities.

  • Retrieval: Video-to-audio and audio-to-video retrieval performed exceptionally well (R@1 ~60%), likely due to the tight synchronization of game events. Text-to-audio proved to be the most challenging retrieval task.
  • Generation: In Text-to-Video (T2V) tasks, human evaluators chose MUGEN-generated videos over ground truth 31.25% of the time for quality and 42.6% for faithfulness.
  • Efficiency: Because the data can be stored as game metadata (JSON) rather than raw video, it is over 100x smaller in storage size than lossless high-resolution video, making it highly accessible for researchers.

A major insight from the experiments was that models often generate "faithful" content (matching the text prompt) even if the visual reconstruction quality is limited by the VQ-VAE bottleneck. This suggests that future improvements in generative architectures, such as Diffusion Models, could further enhance the visual fidelity of MUGEN-based generations.

Conclusion

MUGEN provides a scalable, controllable, and rich playground for the next generation of multimodal models. By bridging the gap between toy datasets and the chaotic real world, it enables researchers to focus on the core challenges of synchronized spatiotemporal generation and understanding. The authors have released the modified game engine and dataset to encourage community expansion into new worlds and interaction types.

Summary updated

Abstract

Multimodal video-audio-text understanding and generation can benefit from datasets that are narrow but rich. The narrowness allows bite-sized challenges that the research community can make progress on. The richness ensures we are making progress along the core challenges. To this end, we present a large-scale video-audio-text dataset MUGEN, collected using the open-sourced platform game CoinRun [11]. We made substantial modifications to make the game richer by introducing audio and enabling new interactions. We trained RL agents with different objectives to navigate the game and interact with 13 objects and characters. This allows us to automatically extract a large collection of diverse videos and associated audio. We sample 375K video clips (3.2s each) and collect text descriptions from human annotators. Each video has additional annotations that are extracted automatically from the game engine, such as accurate semantic maps for each frame and templated textual descriptions. Altogether, MUGEN can help progress research in many tasks in multimodal understanding and generation. We benchmark representative approaches on tasks involving video-audio-text retrieval and generation. Our dataset and code are released at: https://mugen-org.github.io/.
BibTeX
@inproceedings{hayes2022mugen,
  author = {Thomas Hayes and Songyang Zhang and Xi Yin and Guan Pang and Sasha Sheng and Harry Yang and Songwei Ge and Qiyuan Hu and Devi Parikh},
  title = {{MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  doi = {10.1007/978-3-031-20074-8\_25},
}
Figure