TL;DR
The paper introduces Unified Reinforcement and Imitation Learning (RIL), a training framework that allows small, lightweight Vision-Language Models (VLMs) to achieve the performance of massive closed-source models. By combining adversarial imitation learning with reinforcement learning, RIL enables student models to inherit the sophisticated generation styles of large teacher models without requiring slow, verbose "thinking" steps during inference.
What Problem Does This Paper Address?
State-of-the-art Vision-Language Models (VLMs) are incredibly capable but suffer from a massive footprint, often requiring tens or hundreds of billions of parameters. Recent trends to improve smaller models rely on the "think-answer" paradigm (pioneered by models like DeepSeek-R1), where the model generates a lengthy internal monologue before answering. While effective, this drastically increases inference latency and memory consumption, making them impractical for edge devices like smartphones or AR glasses.
Traditional knowledge distillation attempts to shrink these models by matching high-dimensional internal features, but this introduces strict architectural constraints—the student and teacher must share identical tokenizers or embedding strategies. The authors address this bottleneck by asking: Can we train a lightweight VLM to generate high-quality, contextually rich responses purely by learning from the text outputs of large teacher models, without sacrificing inference speed?
Methods
The proposed framework, RIL, elegantly unifies two paradigms: Group Relative Policy Optimization (GRPO) (a resource-efficient reinforcement learning algorithm) and Generative Adversarial Imitation Learning (GAIL).
+-------------------------------------------------------------------+
| RIL Framework |
| |
| +-----------------+ Student Text +------------------+ |
| | Small Student | ---------------------> | LLM-as-a-Judge | |
| | VLM | | (Answer Reward) | |
| +-----------------+ +------------------+ |
| ^ | |
| | v |
| | Combined Reward Signal <------------ [Summation] |
| | ^ |
| +-----------------+ | |
| | Discriminator | <--------------------------------+ |
| | (Style Reward) | <--- Teacher Text (from Large VLMs) |
| +-----------------+ |
+-------------------------------------------------------------------+
The training loop operates on a dual-reward system:
- Similarity (Style) Reward: An LLM-based discriminator is trained to distinguish between student-generated text and teacher-generated text. To prevent training instability, the discriminator's continuous score is binarized (0 or 1), providing a clear signal that forces the student to mimic the teacher's articulation style.
- Answer (Factual) Reward: Because style doesn't guarantee correctness, an LLM-as-a-Judge (specifically Qwen2.5-32B) evaluates the student's response against the ground truth to provide a binary accuracy reward.
Crucially, RIL incorporates text responses from multiple large teacher VLMs (such as Qwen2.5-VL-72B and InternVL3-78B) simultaneously during the reinforcement learning step. This provides a diverse set of high-quality exemplars, stabilizing training and preventing the student from overfitting to a single model's quirks. Because RIL operates entirely on natural language outputs, it is completely agnostic to the underlying tokenizers or image embedding strategies of the teacher models.
Datasets
- Supervised Fine-Tuning (SFT) Phase: A massive 4M-sample visual instruction tuning dataset spanning general VQA, chart/document understanding, math problem-solving, and common-sense reasoning.
- RIL Loop Phase: A curated 40K-sample dataset derived from the SFT pool using log-probability sampling and overlong filtering.
- Evaluation Benchmarks: The models were rigorously tested across 14 diverse benchmarks, including AI2D (diagrams), MathVista (mathematical reasoning), MM-Vet (integrated capabilities), MMMU (expert-level multi-discipline), and BLINK (multimodal perception).
Key Insights and Ideas
- The Multi-Teacher Advantage: Sourcing responses from multiple large teacher VLMs significantly outperforms relying on a single teacher. A diverse pool of teacher responses creates a more robust discriminator and provides the student with richer pathways to find correct answers.
- Binarization Stabilizes Adversarial Training: Instead of using continuous discriminator scores (0.0 to 1.0), converting the score into a binary value (0 or 1) provides a decisive learning signal, mitigating the classic "balance problem" where either the generator or discriminator overpowers the other.
- LLM-as-a-Judge Over Rules-Based Parsing: Traditional exact-match or regex-based answer parsing fails on open-ended visual questions (e.g., "How to cook this dish?"). Utilizing an LLM as a judge allows the framework to scale across a broad spectrum of complex, open-ended tasks.
- Synergy with Distillation: RIL is exceptionally potent when applied to student models that have already undergone feature-level distillation (e.g., MiniLLM, LLaVA-KD), as the prior feature alignment primes them for RIL's linguistic alignment.
Key Results
RIL delivers substantial performance boosts across all tested small model architectures (1B, 2B, 3B, and 7B/8B parameters) without adding a single millisecond of inference latency.
As shown in Figure 1, applying RIL to Qwen2.5-VL-7B yields dramatic improvements compared to standard RL baselines (GRPO and Dr.GRPO), especially when leveraging multiple teachers.
When looking at the scaling curves in Figure 2, the RIL-trained models (like InternVL3-RIL-8B and Qwen2.5-VL-RIL-7B) completely alter the parameter-to-performance tradeoff. They narrow the gap to 70B+ open-source giants and perform competitively with premier closed-source models like GPT-4o and Claude-3.5 Sonnet, all while remaining small enough to run efficiently on consumer edge hardware.
Limitations and Future Work
While RIL shows massive gains, the authors note a few limitations:
- Alignment Phase Focus: The current implementation is strictly focused on the post-instruction tuning alignment phase. Future work will explore integrating the discriminator directly into the initial visual instruction pre-training stage.
- Training Compute Overhead: Training requires running the student, multiple teachers, a discriminator, and an LLM judge simultaneously. Although the authors mitigated this by dynamically offloading weights between CPU and GPU memory, the training footprint remains computationally heavy.
Summary updated