Abstract
Empowering Small-scale Vision-Language Models (SVLMs) with reliable thinking capabilities remains fundamentally challenging due to their limited parameter capacity and weak instruction-following abilities. Existing training paradigms, including Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Reward (RLVR), impose substantial demands on the base VLM, exceeding the capabilities of SVLMs. Consequently, directly applying these paradigms to SVLMs often suffers from severe pseudo thinking traces and advantage collapse, ultimately undermining both thinking reliability and task performance. To address this, we propose DyME, a novel training paradigm that Dynamically selects between Memorization (via SFT) and Exploration (via RLVR) modes at each optimization step, ensuring that every update contributes to the trade-off. Extensive experiments across diverse domains demonstrate that DyME consistently achieves this balance, and thus delivers substantial performance improvements. These results establish DyME as a practical and effective solution for empowering SVLMs with reliable thinking capabilities.
Introduction: The Capacity Gap in SVLMs
While Large Vision-Language Models (LVLMs) like GPT-4o have demonstrated remarkable Chain-of-Thought (CoT) reasoning, Small-scale Vision-Language Models (SVLMs) under 1B parameters struggle to replicate this success. This research identifies a critical "capacity gap" where standard training techniques fail for smaller models.
Supervised Fine-Tuning (SFT) on complex reasoning traces often leads to "pseudo thinking." Instead of grounding their logic in the provided image, SVLMs tend to memorize verbose text patterns, losing visual accuracy. Conversely, Reinforcement Learning with Verifiable Reward (RLVR) requires the model to explore solutions independently. Due to weak instruction-following, SVLMs often fail to produce correctly formatted answers or find the right solution at all, leading to "advantage collapse" where the model receives no meaningful training signal.
The DyME Training Paradigm
To bridge this gap, the researchers introduced DyME (Dynamic Memorize-Explore). Unlike traditional two-stage training (SFT followed by RL), DyME integrates these modes into a single, dynamic optimization process that switches between memorization and exploration at every step.
The Dynamic Switcher
At each training step, the model generates multiple candidate responses for a given input. The system then applies a binary decision rule:
- Exploration Mode (RLVR): If at least one generated response is correct, the model uses RL to reinforce the successful path.
- Memorization Mode (SFT): If all responses are incorrect, the model falls back to SFT, using a high-quality "teacher" trace to correct its path.
This ensures that the model is always learning from a stable signal, preventing the "unconstrained exploration" that typically causes small models to fail.
Visual Supervision and Grounding
To ensure the model's "thinking" is actually grounded in the image, DyME employs a two-part system:
- Visual Refiner: Uses a larger model to restructure raw data into structured, image-grounded reasoning templates.
- Visual Checker: Acts as a critic during RL, rewarding the model not just for the right answer, but for mentioning the correct visual elements (e.g., specific bar values in a chart).
Experimental Results
The DyME paradigm was rigorously tested across diverse domains requiring high-level reasoning, including ChartQA, SLAKE (medical), and MathVerse (geometry). Key findings include:
- Significant Performance Gains: A 500M parameter SmolVLM improved from 49.9% to 55.6% on ChartQA, while standard SFT actually caused performance to degrade.
- Competitive Efficiency: DyME-trained 0.5B models achieved results comparable to much larger, more complex architectures like MoVA.
- Robustness: The "binary switch" mechanism proved more effective than complex annealing schedules. Ablation studies confirmed that the "Memorization" fallback is essential; removing it led to a 20.6% performance drop.
Conclusion
DyME provides a practical pathway for empowering small-scale models with reliable reasoning capabilities. By dynamically balancing memorization and exploration, it overcomes the capacity limitations that previously hindered SVLMs, enabling them to produce grounded, accurate thinking traces without the need for massive parameter counts or expensive proprietary data.
GitHub: https://github.com/HKUST-LongGroup/DyME
Summary updated