Unified Reinforcement and Imitation Learning for Vision-Language Models

Byung-Kwan Lee, Ryo Hachiuma, Yong Man Ro, Yu-Chiang Frank Wang, Yueh-Hua Wu

TL;DR

The paper introduces Unified Reinforcement and Imitation Learning (RIL), a training framework that allows small, lightweight Vision-Language Models (VLMs) to achieve the performance of massive closed-source models. By combining adversarial imitation learning with reinforcement learning, RIL enables student models to inherit the sophisticated generation styles of large teacher models without requiring slow, verbose "thinking" steps during inference.

What Problem Does This Paper Address?

State-of-the-art Vision-Language Models (VLMs) are incredibly capable but suffer from a massive footprint, often requiring tens or hundreds of billions of parameters. Recent trends to improve smaller models rely on the "think-answer" paradigm (pioneered by models like DeepSeek-R1), where the model generates a lengthy internal monologue before answering. While effective, this drastically increases inference latency and memory consumption, making them impractical for edge devices like smartphones or AR glasses.

Traditional knowledge distillation attempts to shrink these models by matching high-dimensional internal features, but this introduces strict architectural constraints—the student and teacher must share identical tokenizers or embedding strategies. The authors address this bottleneck by asking: Can we train a lightweight VLM to generate high-quality, contextually rich responses purely by learning from the text outputs of large teacher models, without sacrificing inference speed?

Methods

The proposed framework, RIL, elegantly unifies two paradigms: Group Relative Policy Optimization (GRPO) (a resource-efficient reinforcement learning algorithm) and Generative Adversarial Imitation Learning (GAIL).

+-------------------------------------------------------------------+
|                           RIL Framework                           |
|                                                                   |
|  +-----------------+      Student Text      +------------------+  |
|  |  Small Student  | ---------------------> |  LLM-as-a-Judge  |  |
|  |       VLM       |                        |  (Answer Reward) |  |
|  +-----------------+                        +------------------+  |
|           ^                                           |           |
|           |                                           v           |
|           | Combined Reward Signal <------------ [Summation]      |
|           |                                           ^           |
|  +-----------------+                                  |           |
|  |  Discriminator  | <--------------------------------+           |
|  | (Style Reward)  | <--- Teacher Text (from Large VLMs)          |
|  +-----------------+                                              |
+-------------------------------------------------------------------+

The training loop operates on a dual-reward system:

  1. Similarity (Style) Reward: An LLM-based discriminator is trained to distinguish between student-generated text and teacher-generated text. To prevent training instability, the discriminator's continuous score is binarized (0 or 1), providing a clear signal that forces the student to mimic the teacher's articulation style.
  2. Answer (Factual) Reward: Because style doesn't guarantee correctness, an LLM-as-a-Judge (specifically Qwen2.5-32B) evaluates the student's response against the ground truth to provide a binary accuracy reward.

Crucially, RIL incorporates text responses from multiple large teacher VLMs (such as Qwen2.5-VL-72B and InternVL3-78B) simultaneously during the reinforcement learning step. This provides a diverse set of high-quality exemplars, stabilizing training and preventing the student from overfitting to a single model's quirks. Because RIL operates entirely on natural language outputs, it is completely agnostic to the underlying tokenizers or image embedding strategies of the teacher models.

Datasets

  • Supervised Fine-Tuning (SFT) Phase: A massive 4M-sample visual instruction tuning dataset spanning general VQA, chart/document understanding, math problem-solving, and common-sense reasoning.
  • RIL Loop Phase: A curated 40K-sample dataset derived from the SFT pool using log-probability sampling and overlong filtering.
  • Evaluation Benchmarks: The models were rigorously tested across 14 diverse benchmarks, including AI2D (diagrams), MathVista (mathematical reasoning), MM-Vet (integrated capabilities), MMMU (expert-level multi-discipline), and BLINK (multimodal perception).

Key Insights and Ideas

  • The Multi-Teacher Advantage: Sourcing responses from multiple large teacher VLMs significantly outperforms relying on a single teacher. A diverse pool of teacher responses creates a more robust discriminator and provides the student with richer pathways to find correct answers.
  • Binarization Stabilizes Adversarial Training: Instead of using continuous discriminator scores (0.0 to 1.0), converting the score into a binary value (0 or 1) provides a decisive learning signal, mitigating the classic "balance problem" where either the generator or discriminator overpowers the other.
  • LLM-as-a-Judge Over Rules-Based Parsing: Traditional exact-match or regex-based answer parsing fails on open-ended visual questions (e.g., "How to cook this dish?"). Utilizing an LLM as a judge allows the framework to scale across a broad spectrum of complex, open-ended tasks.
  • Synergy with Distillation: RIL is exceptionally potent when applied to student models that have already undergone feature-level distillation (e.g., MiniLLM, LLaVA-KD), as the prior feature alignment primes them for RIL's linguistic alignment.

Key Results

RIL delivers substantial performance boosts across all tested small model architectures (1B, 2B, 3B, and 7B/8B parameters) without adding a single millisecond of inference latency.

Figure 1: Showing the performance improvements (%) of Qwen2.5-VL-7B [1] across vision-language evaluation benchmarks for AI2D [2], MathVista [3], MM-Vet [4], MMMU [5], BLINK [6], and the average scores for 14 evaluation benchmarks used in Table 1. Note that, we conduct RL on GRPO [7] and advanced GRPO, Dr.GRPO [8], with only answer rewards from LLM-as-a-Judge [9] (see Algorithm 1), and we present RIL based on similarity rewards from single or multi large teacher VLMs and simultaneously answer rewards (see Algorithm 2).

As shown in Figure 1, applying RIL to Qwen2.5-VL-7B yields dramatic improvements compared to standard RL baselines (GRPO and Dr.GRPO), especially when leveraging multiple teachers.

Figure 2: Comparing RIL-applied VLMs based on multi large VLMs with diverse open- and closed- source VLMs, under average performance of numerous vision-language evaluation benchmarks: AI2D [2], ChartQA [19], MathVista [3], MMB [20], MM-Vet [4], MMMU [5], MMMU-Pro [21], MMStar [22], BLINK [6], SEED [23], SEED2+ [24], and RealWorldQA (RWQA).

When looking at the scaling curves in Figure 2, the RIL-trained models (like InternVL3-RIL-8B and Qwen2.5-VL-RIL-7B) completely alter the parameter-to-performance tradeoff. They narrow the gap to 70B+ open-source giants and perform competitively with premier closed-source models like GPT-4o and Claude-3.5 Sonnet, all while remaining small enough to run efficiently on consumer edge hardware.

Limitations and Future Work

While RIL shows massive gains, the authors note a few limitations:

  • Alignment Phase Focus: The current implementation is strictly focused on the post-instruction tuning alignment phase. Future work will explore integrating the discriminator directly into the initial visual instruction pre-training stage.
  • Training Compute Overhead: Training requires running the student, multiple teachers, a discriminator, and an LLM judge simultaneously. Although the authors mitigated this by dynamically offloading weights between CPU and GPU memory, the training footprint remains computationally heavy.

Summary updated

Abstract

Vision-Language Models (VLMs) have achieved remarkable progress, yet their large scale often renders them impractical for resource-constrained environments. This paper introduces Unified Reinforcement and Imitation Learning (RIL), a novel and efficient training algorithm designed to create powerful, lightweight VLMs. RIL distinctively combines the strengths of reinforcement learning with adversarial imitation learning. This enables smaller student VLMs not only to mimic the sophisticated text generation of large teacher models but also to systematically improve their generative capabilities through reinforcement signals. Key to our imitation framework is an LLM-based discriminator that adeptly distinguishes between student and teacher outputs, complemented by guidance from multiple large teacher VLMs to ensure diverse learning. This unified learning strategy, leveraging both reinforcement and imitation, empowers student models to achieve significant performance gains, making them competitive with leading closed-source VLMs. Extensive experiments on diverse vision-language benchmarks demonstrate that RIL significantly narrows the performance gap with state-of-the-art open- and closed-source VLMs and, in several instances, surpasses them.
BibTeX
@inproceedings{lee2025unified,
  author = {Byung-Kwan Lee and Ryo Hachiuma and Yong Man Ro and Yu-Chiang Frank Wang and Yueh-Hua Wu},
  title = {{Unified Reinforcement and Imitation Learning for Vision-Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=7wEvjzkNXg},
}
Figure