Abstract
Unified Multimodal Models (UMMs) have shown impressive performance in both understanding and generation with a single architecture. However, UMMs still exhibit a fundamental inconsistency: understanding favors compact embeddings, whereas generation favors reconstruction-rich representations. This structural trade-off produces misaligned decision boundaries, degraded cross-modal coherence, and heightened vulnerability under distributional and adversarial shifts. In this paper, we present UniGame, a self-adversarial post-training framework that directly targets the inconsistencies. By applying a lightweight perturber at the shared token interface, UniGame enables the generation branch to actively seek and challenge fragile understanding, turning the model itself into its own adversary. Experiments demonstrate that UniGame significantly improves the consistency (+4.6%). Moreover, it also achieves substantial improvements in understanding (+3.6%), generation (+0.02), out-of-distribution and adversarial robustness (+4.8% and +6.2% on NaturalBench and AdVQA). The framework is architecture-agnostic, introduces less than 1% additional parameters, and is complementary to existing post-training methods. These results position adversarial self-play as a general and effective principle for enhancing the coherence, stability, and unified competence of future multimodal foundation models. The official code is available at: https://github.com/AIFrontierLab/UniGame
Introduction: The Identity Crisis of Unified Models
Unified Multimodal Models (UMMs), such as Janus or Emu3, represent the frontier of AI by integrating both understanding ("seeing") and generation ("drawing") within a single architecture. Despite their versatility, these models suffer from a fundamental structural tension. Understanding tasks prioritize "compact" representations that distill an image into its core semantic meaning. Conversely, generation tasks require "reconstruction-rich" representations that preserve granular pixel details.
This divergence leads to significant structural inconsistency. A model may accurately identify an object in a scene yet fail to generate a coherent image of that same object in a similar context. Traditional post-training methods, such as Supervised Fine-Tuning (SFT), often fail to address these internal "blind spots" where the understanding and generation pathways are misaligned.
Methodology: UniGame Self-Adversarial Framework
To resolve these inconsistencies, we introduce UniGame, a post-training framework based on a minimax game. UniGame transforms the UMM into its own "sparring partner," using the generation branch to challenge the understanding branch.
The framework consists of several key components:
- The Perturber ($C$): A lightweight, three-layer MLP that introduces calculated "nudges" (perturbations) to visual tokens at the shared interface.
- Adversarial Generation: These perturbed tokens are processed by the model's own image decoder. Because the perturbations are constrained, the resulting images are realistic "counterexamples" designed to expose the understanding branch's weaknesses.
- Hard-Sample Buffer ($\mathcal{B}$): Samples that successfully deceive the understanding branch while maintaining semantic validity (verified via CLIP similarity) are stored for training.
- Minimax Optimization: The understanding branch is trained on a combination of clean data and these self-generated "hard" samples. This forces the model to reconcile its internal representations and become more robust.
By constraining the adversary through the model's own decoder, UniGame ensures that the generated challenges remain "on-manifold"---visually and semantically meaningful rather than abstract mathematical noise.
Evaluation and Results
We evaluated UniGame across a comprehensive suite of benchmarks focusing on understanding (MMMU, POPE), generation (GenEval), consistency (UnifiedBench, WISE), and robustness (NaturalBench, AdVQA).
Key Findings
- Enhanced Consistency: UniGame achieved a +4.6% improvement in consistency, indicating better alignment between understanding and generation branches.
- Superior Understanding: The model showed a +3.6% boost on the college-level reasoning benchmark (MMMU).
- Robustness Gains: Significant improvements were observed in handling out-of-distribution images (+4.8% on NaturalBench) and resisting adversarial linguistic traps (+6.2% on AdVQA).
- Manifold Expansion: As shown in the manifold coverage analysis, UniGame effectively "inflates" the model's training distribution, allowing it to explore and master realistic edge cases that are typically absent from standard datasets.
Discussion and Future Work
UniGame demonstrates that self-play, a principle that revolutionized games like Go, is equally effective for refining multimodal foundation models. By turning a model into its own adversary, we can force the resolution of internal structural tensions without requiring massive new datasets.
While the current study primarily utilized the Janus-Pro-7B architecture and specific training sets (VQAv2, CC3M), the framework is architecture-agnostic and parameter-efficient (adding <1% parameters). Future research will focus on scaling UniGame to larger models and more diverse data distributions to further explore the limits of self-adversarial multimodal learning.
Summary updated