Abstract
Vision Language Models (VLMs) exhibit impressive performance for various tasks, yet they often lack the sophisticated situational reasoning required for complex decision-making. This paper shows that VLMs can achieve surprisingly strong decision-making performance when visual scenes are replaced by textual descriptions, suggesting foundational reasoning can be effectively learned from language. Motivated by this insight, we propose Praxis-VLM, a reasoning VLM for vision-grounded decision-making. Praxis-VLM employs the GRPO algorithm on textual scenarios to instill robust reasoning capabilities, where models learn to evaluate actions and their consequences. These reasoning skills, acquired purely from text, successfully transfer to multimodal inference with visual inputs, significantly reducing reliance on scarce paired image-text training data. Experiments across diverse decision-making benchmarks demonstrate that Praxis-VLM substantially outperforms standard supervised fine-tuning, exhibiting superior performance and generalizability.
Introduction
Modern Vision Language Models (VLMs) are adept at visual perception but frequently falter when faced with the multi-step situational reasoning required for real-world decision-making, such as in robotics or interactive AI assistance. Traditionally, teaching these models to reason has required massive datasets of paired images and step-by-step textual explanations—data that is notoriously difficult and expensive to collect.
Praxis-VLM addresses this bottleneck by exploring whether decision-making ability is exclusively tied to direct multimodal experience. The authors hypothesize that foundational reasoning can be disentangled from visual perception, learned primarily through language, and then successfully applied to visually grounded contexts.
Methodology: Text-Driven Reinforcement Learning
The core innovation of Praxis-VLM is a text-driven Reinforcement Learning (RL) paradigm. Instead of training on image-text pairs, the model's reasoning components are trained on text-only scenarios, while inference is performed on full multimodal inputs.
The training process involves three key steps:
- Synthetic Data Generation: Using GPT-4o, the authors created a dataset of 10,000 text-only situational descriptions paired with multiple-choice decision-making questions.
- Multi-Stage GRPO Training: The VLM (Qwen2.5-VL) is optimized using Group Relative Policy Optimization (GRPO). A "cold start" on mathematical logic (
geometry3k) establishes basic logical structuring. This is followed by decision-making RL on the synthetic scenarios, utilizing a novel length reward to encourage deliberate, multi-step reasoning chains. - Multimodal Transfer: During inference, the model applies its text-learned reasoning skills to analyze actual images or videos, making decisions based on visual evidence without having seen similar images during the reasoning training phase.
Experimental Results
Praxis-VLM was evaluated across three major benchmarks: VIVA (social situations), PCA-Bench (robotics and driving), and EgoNormia (ego-centric video).
- Performance Gains: Praxis-VLM-7B achieved 84.03% accuracy on VIVA and 60.25% on PCA-Bench, significantly outperforming the vanilla Qwen2.5-VL-7B.
- Generalization: On the out-of-domain EgoNormia video benchmark, Praxis-VLM scored 54.33%, vastly exceeding the Supervised Fine-Tuning (SFT) baseline of 34.83%.
- Emergent Reasoning: The model naturally learned to consider situational analysis, consequence evaluation, and safety management in its reasoning paths, despite no explicit instructions to do so.
Conclusion
Praxis-VLM demonstrates that VLMs do not need to "see" to learn how to "think." By leveraging the rich logical structures inherent in language through reinforcement learning, we can equip models with robust decision-making capabilities that generalize across modalities. This approach significantly reduces the need for expensive multimodal reasoning data and paves the way for more capable, reasoning-driven AI agents.
Summary updated