LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL

TL;DR

This paper introduces LMM-R1, a novel two-stage framework that significantly enhances the reasoning capabilities of small (3B parameter) Large Multimodal Models (LMMs). By first strengthening foundational reasoning with text-only data and then generalizing to multimodal tasks, LMM-R1 achieves substantial performance gains without requiring extensive high-quality multimodal training data.

What Problem Does This Paper Address?

Large Multimodal Models (LMMs) are powerful tools that combine visual and textual understanding, but enhancing their reasoning abilities, especially in smaller models (like 3B parameters), is a significant challenge. This is due to:

1. Architectural Constraints: Smaller models have limited capacity, making it hard to integrate complex visual and logical reasoning.
2. Data Scarcity: High-quality multimodal reasoning data is difficult to obtain, often featuring ambiguous answers or lacking complex reasoning examples.
3. Multimodal Pre-training Degradation: Ironically, pre-training on multimodal data can sometimes weaken a model's foundational reasoning abilities, particularly in text-only tasks.

Existing methods like Monte Carlo Tree Search or Process Reward Models often require extensive human annotation or costly computational resources. While rule-based Reinforcement Learning (RL) has shown promise in text-only domains, its direct application to LMMs faces these data and degradation issues.

Methods

LMM-R1 proposes a two-stage rule-based RL approach:

1. Foundational Reasoning Enhancement (FRE): This initial stage focuses on building a strong reasoning foundation. It uses rule-based RL with abundant, high-quality text-only reasoning data. The key idea here is that robust text-based reasoning skills can be effectively transferred to multimodal contexts.
2. Multimodal Generalization Training (MGT): After FRE, this stage generalizes the enhanced reasoning capabilities to various multimodal domains. This involves continuing rule-based RL training on a curated set of complex multimodal reasoning tasks, covering both general reasoning and agent-related scenarios.

The core of the method is rule-based RL, which uses a reward function that encourages both correct formatting (e.g., using <think> and <answer> tags) and accurate solutions. This approach allows models to autonomously improve reasoning through exploration while avoiding reward hacking.

Datasets

The paper utilizes several datasets for training and evaluation:

• Text-Only RL Dataset: DeepScaler-Preview-Dataset (40k math prompt-answer pairs) for FRE-Text.
• Verifiable Multimodal-65K (VerMulti): A filtered subset of MathV360K (130k multimodal problems) for FRE-Multi. This dataset includes arithmetic, geometry, calculus, and science problems.
• VerMulti-Geo: A subset of VerMulti focusing on geometry problems (15k) for MGT-Geo.
• VerMulti: Used directly for the Perception-Reasoning Balanced Domain in MGT.
• Sokoban Training Datasets: 11,500 environments generated for Sokoban to train MGT-Sokoban.
• Evaluation Benchmarks:
  • Text-Only: MATH500, GPQA.
• Multimodal Reasoning-Dominated: OlympiadBench, MathVision, MathVerse.
• General Multimodal: MM-Star, MathVista.
• Agent-Related: Sokoban, Football Game tasks.

Key Insights and Ideas

• Text-Only RL is Crucial for Multimodal Reasoning: The most significant insight is that strengthening foundational reasoning using text-only data via rule-based RL is more effective for multimodal generalization than directly training on multimodal data. This is counter-intuitive, as one might expect direct multimodal training to be superior.
• Avoiding Catastrophic Forgetting: The FRE stage, by focusing on text-only data, helps prevent the degradation of pure reasoning abilities that can occur when models are trained solely on multimodal data.
• Compact Models Can Achieve Strong Reasoning: The paper demonstrates that even small 3B LMMs can achieve impressive reasoning capabilities with the proposed two-stage approach, overcoming their inherent capacity limitations.
• Synergistic Effect of Two-Stage Training: The FRE stage provides a robust foundation that makes the subsequent MGT stage more effective, leading to better performance across diverse multimodal tasks.

Key Results

LMM-R1, applied to the Qwen2.5-VL-Instruct-3B model, achieved significant improvements:

• Average Improvement: 4.83% on multimodal benchmarks and 4.5% on text-only benchmarks compared to baselines.
• Complex Tasks: A 3.63% gain on complex Football Game tasks.
• FRE-Text vs. FRE-Multi: FRE-Text (text-only FRE) showed substantial improvements on text-only benchmarks (e.g., +6.57% on GPQA) and strong generalization to multimodal reasoning-dominated tasks. FRE-Multi (multimodal FRE) showed better performance on general visual tasks but declined on text-only reasoning.
• Agent Domains: The approach demonstrated effective generalization to agent-related tasks like Sokoban, matching or surpassing product-level large models.

Limitations and Future Work

• Limitations:
  • The paper notes that while FRE-Text improves reasoning, it might slightly compromise visual perception capabilities, which MGT-Geo helps to recover.
• The performance gains are most pronounced in smaller 3B LMMs, suggesting that larger models might already possess stronger foundational reasoning.
• Future Work:
  • Extending the LMM-R1 framework to additional LMM architectures.
• Developing methods for synthesizing high-quality multimodal reasoning data for rule-based RL, which could further improve efficiency and effectiveness.