Praxis-VLM: Vision-Grounded Decision Making via Text-Driven Reinforcement Learning

Zhe Hu, Jing Li, Zhongzhu Pu, Hou Pong Chan, Yu Yin

Abstract

Vision Language Models (VLMs) exhibit impressive performance for various tasks, yet they often lack the sophisticated situational reasoning required for complex decision-making. This paper shows that VLMs can achieve surprisingly strong decision-making performance when visual scenes are replaced by textual descriptions, suggesting foundational reasoning can be effectively learned from language. Motivated by this insight, we propose Praxis-VLM, a reasoning VLM for vision-grounded decision-making. Praxis-VLM employs the GRPO algorithm on textual scenarios to instill robust reasoning capabilities, where models learn to evaluate actions and their consequences. These reasoning skills, acquired purely from text, successfully transfer to multimodal inference with visual inputs, significantly reducing reliance on scarce paired image-text training data. Experiments across diverse decision-making benchmarks demonstrate that Praxis-VLM substantially outperforms standard supervised fine-tuning, exhibiting superior performance and generalizability.

Introduction

Modern Vision Language Models (VLMs) are adept at visual perception but frequently falter when faced with the multi-step situational reasoning required for real-world decision-making, such as in robotics or interactive AI assistance. Traditionally, teaching these models to reason has required massive datasets of paired images and step-by-step textual explanations—data that is notoriously difficult and expensive to collect.

A complex visual scene, such as a grocery shelf, requires sophisticated reasoning to navigate.

Praxis-VLM addresses this bottleneck by exploring whether decision-making ability is exclusively tied to direct multimodal experience. The authors hypothesize that foundational reasoning can be disentangled from visual perception, learned primarily through language, and then successfully applied to visually grounded contexts.

Methodology: Text-Driven Reinforcement Learning

The core innovation of Praxis-VLM is a text-driven Reinforcement Learning (RL) paradigm. Instead of training on image-text pairs, the model's reasoning components are trained on text-only scenarios, while inference is performed on full multimodal inputs.

The Praxis-VLM framework emphasizes text-driven training to develop transferable reasoning skills.

The training process involves three key steps:

  1. Synthetic Data Generation: Using GPT-4o, the authors created a dataset of 10,000 text-only situational descriptions paired with multiple-choice decision-making questions.
  2. Multi-Stage GRPO Training: The VLM (Qwen2.5-VL) is optimized using Group Relative Policy Optimization (GRPO). A "cold start" on mathematical logic (geometry3k) establishes basic logical structuring. This is followed by decision-making RL on the synthetic scenarios, utilizing a novel length reward to encourage deliberate, multi-step reasoning chains.
  3. Multimodal Transfer: During inference, the model applies its text-learned reasoning skills to analyze actual images or videos, making decisions based on visual evidence without having seen similar images during the reasoning training phase.

Experimental Results

Praxis-VLM was evaluated across three major benchmarks: VIVA (social situations), PCA-Bench (robotics and driving), and EgoNormia (ego-centric video).

  • Performance Gains: Praxis-VLM-7B achieved 84.03% accuracy on VIVA and 60.25% on PCA-Bench, significantly outperforming the vanilla Qwen2.5-VL-7B.
  • Generalization: On the out-of-domain EgoNormia video benchmark, Praxis-VLM scored 54.33%, vastly exceeding the Supervised Fine-Tuning (SFT) baseline of 34.83%.
  • Emergent Reasoning: The model naturally learned to consider situational analysis, consequence evaluation, and safety management in its reasoning paths, despite no explicit instructions to do so.

Conclusion

Praxis-VLM demonstrates that VLMs do not need to "see" to learn how to "think." By leveraging the rich logical structures inherent in language through reinforcement learning, we can equip models with robust decision-making capabilities that generalize across modalities. This approach significantly reduces the need for expensive multimodal reasoning data and paves the way for more capable, reasoning-driven AI agents.

Summary updated

Abstract

Vision Language Models exhibit impressive performance for various tasks, yet they often lack the sophisticated situational reasoning required for complex decision-making. This paper shows that VLMs can achieve surprisingly strong decision-making performance when visual scenes are replaced by textual descriptions, suggesting foundational reasoning can be effectively learned from language. Motivated by this insight, we propose Praxis-VLM, a reasoning VLM for vision-grounded decision-making. Praxis-VLM employs the GRPO algorithm on textual scenarios to instill robust reasoning capabilities, where models learn to evaluate actions and their consequences. These reasoning skills, acquired purely from text, successfully transfer to multimodal inference with visual inputs, significantly reducing reliance on scarce paired image-text training data. Experiments across diverse decision-making benchmarks demonstrate that Praxis-VLM substantially outperforms standard supervised fine-tuning, exhibiting superior performance and generalizability. Further analysis confirms that our models engage in explicit and effective reasoning, underpinning their enhanced performance and adaptability.
BibTeX
@inproceedings{hu2025praxis,
  author = {Zhe Hu and Jing Li and Zhongzhu Pu and Hou Pong Chan and Yu Yin},
  title = {{Praxis-VLM: Vision-Grounded Decision Making via Text-Driven Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=U806q3iILo},
}
Figure