Dynamic Adversarial Reinforcement Learning for Robust Multimodal Large Language Models

Yicheng Bao, Xuhong Wang, Qiaosheng Zhang, Chaochao Lu, Xia Hu, Xin TanarXiv 2026 · arXiv

TL;DR

Multimodal Large Language Models (MLLMs) are notoriously fragile when faced with complex scenes or unexpected visual distractors. To solve this, researchers have introduced Adversarial Opponent Training (AOT), a self-play framework where an image-editing "Attacker" and an MLLM "Defender" continuously co-evolve. By generating a dynamic curriculum of semantic image manipulations, this framework significantly boosts MLLM perceptual robustness, reduces hallucinations, and improves general visual reasoning.

Figure 1. An illustration of the perceptual fragility in existing MLLMs and the robustness fostered by our co-evolutionary framework. Left (Co-evolution Concept): A conceptual depiction of the iterative competition between an Attacker and a Defender model, where each model’s capabilities are progressively enhanced. Right (Practical Demonstration): In a simple scene, both a standard MLLM and our Defender can correctly identify the spatial relation, but our model demonstrates superior, detailed perception. When a contextual distractor is introduced, the standard MLLM is perceptually misled by the distractor, causing its subsequent reasoning to fail. In contrast, our Defender, which shares the same base MLLM architecture, grounds its reasoning in a robust perceptual understanding of the scene, a capability directly fostered by our adversarial training process.

What Problem Does This Paper Address?

While modern MLLMs display impressive reasoning capabilities, their underlying visual perception remains surprisingly brittle. A minor modification to an image—such as introducing an irrelevant object (a distractor) near a target item—can completely derail a model's spatial understanding, leading to incorrect answers and hallucinations.

The root cause lies in the current training paradigm: MLLMs rely heavily on finite, static datasets. Manually annotating fine-grained perceptual data is prohibitively expensive, and static adversarial datasets quickly become obsolete as models improve. To break through this capability ceiling, this paper shifts the paradigm from fixed corpora to autonomous, dynamic data generation via adversarial self-play.

Methods

The core of the paper is Adversarial Opponent Training (AOT), a self-play framework that orchestrates an iterative game between two models:

  1. The Attacker ($M_{atk}$): An image-editing model (based on Qwen Image Edit) optimized using Flow-GRPO (a reinforcement learning algorithm for generative models). Its goal is to execute localized semantic edits—like adding, removing, or replacing objects—to trick the defender. The attacker's reward is tied to both adversarial efficacy (successfully fooling the defender twice consecutively) and semantic integrity (ensuring it doesn't corrupt the core question area, verified via a localized Structural Similarity Index Measure, or SSIM).
  2. The Defender ($M_{def}$): The target MLLM (built on Qwen2.5-VL-7B) trained via DAPO on the attacker's generated images.

To prevent the framework from stalling due to a "cold start" (where the base image editor fails to comprehend complex adversarial instructions), the authors first design a rigorous two-stage data generation pipeline to create an initial dataset (AOT-SFT) for bootstrapping the attacker.

Demonstration of why initial SFT is necessary: un-tuned base models fail to generate proper distractors.

Datasets

  • AOT-SFT: A newly introduced large-scale adversarial dataset featuring paired clean and adversarially manipulated high-resolution images, used to bootstrap the self-play framework. It builds upon the VStar spatial relation dataset by outpainting scenes to increase complexity and implanting validated semantic distractors.
  • Evaluation Benchmarks:
  • Perceptual Robustness: VStar and HRBench (4K and 8K variants).
  • Factuality & Hallucination: POPE and HallusionBench.
  • General Capabilities: MMMU, MMStar, RealWorldQA, BLINK, and AI2D (to monitor for catastrophic forgetting).

Key Insights and Ideas

  • Pixel-Level Self-Play: Unlike traditional LLM self-play frameworks that operate entirely within text spaces, AOT directly manipulates image pixels. This forces the MLLM to ground its reasoning in robust visual perception rather than relying on language heuristics.
  • Emergent Attack Taxonomy: Although the attacker was initially fine-tuned only on object addition examples, during the co-evolutionary process it autonomously discovered a diverse set of advanced strategies, including object removal, replacement, and hybrid manipulations.
  • The "Just Right" Curriculum: The authors discovered that training a defender on only the hardest adversarial examples severely harms performance. Instead, they implement a "difficulty windowing" strategy, filtering the training set to include only examples where the defender succeeds between 30% and 70% of the time, targeting the exact boundaries of model capability.
Qualitative examples of diverse attack strategies autonomously discovered by the attacker model.

Key Results

  • Substantial Robustness Gains: After three co-evolutionary iterations, the Defender model achieved 80.25% on VStar (+9.24 points over the base model) and 72.38% on HRBench-4K (+8.26 points). Robustness also generalized to ultra-high-resolution inputs, boosting HRBench-8K scores from 64.88% to 71.50%.
  • Mitigated Hallucinations: The final Defender increased its POPE F1-score by +2.88 points and HallusionBench accuracy by +1.68 points, outperforming all static data augmentation baselines.
  • No Catastrophic Forgetting: Rather than degrading general performance, the robustness training actually enhanced core capabilities, yielding a +4.66 point gain on MMMU Dev and +2.36 points on RealWorldQA.
  • Curriculum Transferability: The adversarial curriculum generated by the Qwen2.5-VL (7B) framework successfully transferred to other model families and scales, boosting Qwen3-VL (8B) by +7.56 points on VStar and Gemma-3-IT (27B) by +5.37 points on HRBench-4K.

Limitations and Future Work

  • Reliance on Objective Ground Truth: The framework currently depends on Visual Question Answering (VQA) tasks with clear, objective correctness criteria. Extending this self-play paradigm to open-ended generative tasks where deception is subjective remains a significant open challenge.
  • Computational Efficiency: Iterative reinforcement learning loops across multimodal models are computationally heavy. Future work will explore more efficient co-evolutionary training schemes and methods to further enhance the attacker's generative diversity.

Summary updated

Abstract

Despite their impressive capabilities, Multimodal Large Language Models (MLLMs) exhibit perceptual fragility when confronted with visually complex scenes. This weakness stems from a reliance on finite training datasets, which are prohibitively expensive to scale and impose a ceiling on model robustness. We introduce \textbf{AOT-SFT}, a large-scale adversarial dataset for bootstrapping MLLM robustness. Building on this, we propose \textbf{AOT (Adversarial Opponent Training)}, a self-play framework that forges MLLM robustness by creating its own training data. Our method orchestrates a co-evolution between an image-editing Attacker and a Defender MLLM, where the Attacker generates a diverse and dynamic curriculum of image manipulations, forcing the Defender to adapt and improve. Extensive experiments demonstrate that AOT enhances the Defender's perceptual robustness and reduces hallucinations, establishing a scalable paradigm for training more reliable MLLMs.
BibTeX
@misc{bao2026dynamic,
  author = {Yicheng Bao and Xuhong Wang and Qiaosheng Zhang and Chaochao Lu and Xia Hu and Xin Tan},
  title = {{Dynamic Adversarial Reinforcement Learning for Robust Multimodal Large Language Models}},
  year = {2026},
  eprint = {2602.22227},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.22227},
}
Figure