Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme

Yan Ma, Steffi Chern, Xuyang Shen, Yiran Zhong, Pengfei LiuarXiv 2025 · OpenReview

Abstract

Reinforcement learning (RL) has recently shown strong potential in improving the reasoning capabilities of large language models and is now being actively extended to vision-language models (VLMs). However, existing RL applications in VLMs often rely on heavily engineered frameworks that hinder reproducibility and accessibility, while lacking standardized evaluation protocols, making it difficult to compare results or interpret training dynamics. This work introduces a transparent, from-scratch framework for RL in VLMs, offering a minimal yet functional four-step pipeline validated across multiple models and datasets. In addition, a standardized evaluation scheme is proposed to assess training dynamics and reflective behaviors. Extensive experiments on visual reasoning tasks uncover key empirical findings: response length is sensitive to random seeds, reflection correlates with output length, and RL consistently outperforms supervised fine-tuning (SFT) in generalization, even with high-quality data. These findings, together with the proposed framework, aim to establish a reproducible baseline and support broader engagement in RL-based VLM research.

Introduction

While Large Language Models (LLMs) have experienced significant performance leaps through reinforcement learning (RL)—often referred to as the "R1 moment"—extending these successes to Vision Language Models (VLMs) remains challenging. Current research faces two primary hurdles: excessive engineering complexity in existing RL libraries (like TRL or OpenRLHF) and a lack of standardized evaluation protocols for tracking training dynamics beyond final accuracy scores. This paper introduces MAYE, a minimalist, transparent framework designed to lower the barrier to entry for VLM-RL research and provide deeper insights into how these models learn to reason visually.

The MAYE Framework

The MAYE framework is a from-scratch implementation using standard libraries such as Hugging Face Transformers, FSDP2, and vLLM. It avoids the "black box" nature of complex frameworks by utilizing a clear four-step pipeline.

The MAYE framework pipeline illustrating the four-step process from data flow to policy update.

The pipeline consists of:

  1. Step I: Data Flow: Converting multimodal data into model-compatible inputs.
  2. Step II: Response Collection: Utilizing an inference engine to gather model outputs efficiently across multiple GPUs.
  3. Step III: Trajectory Generation: Organizing inputs, outputs, and rule-based rewards into structured trajectories.
  4. Step IV: Policy Update: Updating the model using Reinforce++, which incorporates a KL divergence penalty to ensure training stability.

To maintain efficiency, the framework focuses on training the LLM backbone while keeping the Vision Transformer (ViT) and connector frozen.

Experimental Setup

The study evaluates visual reasoning across two primary modalities using verifiable mathematical tasks:

  • mm_math5k: A text-dominant dataset where images provide supporting context.
  • geometry3k: A vision-dominant dataset requiring direct information extraction from images.
  • MathVerse: Employed for out-of-distribution (OOD) generalization testing.

Key Insights and Results

The application of RL to VLMs reveals several critical training dynamics. A notable phenomenon is the emergence of "reflective" behaviors, where models naturally begin to use self-correction terms (e.g., "re-evaluate", "wait") as training progresses.

Training dynamics across different models and datasets, showing accuracy, response length, and reflection metrics.

Reflection and Response Length

Experiments show a strong correlation between response length and the frequency of reflective reasoning. However, accuracy improvements are not solely tied to increased length; models often become more efficient in their reasoning paths over time. Interestingly, newer models like Qwen2.5-VL exhibit higher baseline reflection compared to their predecessors, suggesting that these capabilities may be partially inherited from advanced pre-training.

RL vs. Supervised Fine-Tuning (SFT)

A major finding is that RL consistently outperforms SFT in generalization. While SFT models often "memorize" the style of training data—sometimes leading to performance degradation on unseen datasets—RL models learn the underlying logical structures required for problem-solving. On average, RL achieved a 1.35x to 1.36x increase in accuracy across the tested datasets.

Sensitivity to Random Seeds

The study highlights the inherent fragility of RL training. Metrics such as response length and overall stability are highly sensitive to random seeds. This underscores the importance of standardized evaluation and reporting averages across multiple runs to ensure scientific rigor.

Conclusion

The MAYE framework provides a reproducible baseline for RL in VLMs, demonstrating that RL significantly enhances visual reasoning and OOD generalization. While currently focused on verifiable tasks like math and geometry, future work will explore scaling these methods to subjective visual tasks and more complex architectures. By offering a transparent and accessible pipeline, this work aims to foster broader community engagement in the evolving field of multimodal reinforcement learning.

Summary updated

Abstract

Reinforcement learning (RL) has recently shown strong potential in improving the reasoning capabilities of large language models and is now being actively extended to vision-language models (VLMs). However, existing RL applications in VLMs often rely on heavily engineered frameworks that hinder reproducibility and accessibility, while lacking standardized evaluation protocols, making it difficult to compare results or interpret training dynamics. This work introduces a transparent, from-scratch framework for RL in VLMs, offering a minimal yet functional four-step pipeline validated across multiple models and datasets. In addition, a standardized evaluation scheme is proposed to assess training dynamics and reflective behaviors. Extensive experiments on visual reasoning tasks uncover key empirical findings: response length is sensitive to random seeds, reflection correlates with output length, and RL consistently outperforms supervised fine-tuning (SFT) in generalization, even with high-quality data. These findings, together with the proposed framework, aim to establish a reproducible baseline and support broader engagement in RL-based VLM research.
BibTeX
@misc{ma2025rethinking,
  author = {Yan Ma and Steffi Chern and Xuyang Shen and Yiran Zhong and Pengfei Liu},
  title = {{Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme}},
  year = {2025},
  eprint = {2504.02587},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.02587},
}
Figure