TL;DR
Reframe is a novel post-training framework designed to fix "observation neglect" in visual agents—where models invoke tools (like image cropping) but ignore the results. By treating the observation state as an object of alignment through a two-stage process of specialized reinforcement learning and "observation-gated" distillation, Reframe produces agents that significantly outperform existing methods across 14 image and video benchmarks.
What Problem Does This Paper Address?
Equipping Vision-Language Models (VLMs) with tools—such as the ability to crop an image for better detail or select specific video frames—is essential for complex reasoning. However, current "agentic" models suffer from two major flaws:
- Observation Neglect: A model might call a tool to see a detail better, but its final answer remains unchanged regardless of what the tool returns. It "looks" but doesn't "see."
- Supervision Sparsity: In standard Reinforcement Learning (RL), if a model looks at the wrong part of an image, it gets a low reward, but it doesn't learn how to reason about the correct part. Conversely, in standard distillation, if a student model looks at the wrong thing, the teacher's feedback is useless because it's conditioned on a "garbage" observation.
The authors argue that we need to align not just the model's output, but the observation state itself, ensuring the model both acquires valid evidence and relies on it.
Methods
Reframe introduces a two-stage training pipeline that bypasses the need for a task-specific Supervised Fine-Tuning (SFT) "cold start."
Stage 1: Ground-Drop-GRPO (GD-GRPO)
This stage trains modality-specific specialists (one for images, one for videos). It uses two clever tricks to force the model to use its tools:
- Original-Observation Dropout: During training, after a model successfully calls a tool (e.g., crops an image), the original full-sized image is stochastically removed. This forces the model to rely entirely on the tool-returned observation to answer the question.
- Evidence-Aware Rewards: Instead of just rewarding the final correct answer, the model is rewarded for the quality of its "grounding" (how well its crop/selection overlaps with known important regions).
Stage 2: Observation-Gated MOPD (OG-MOPD)
This stage consolidates the specialists into a single unified model using Multi-Teacher On-Policy Distillation.
- Observation Gating: If the student model makes a poor tool call (e.g., crops a blank area), the framework intervenes mid-rollout, replaces the bad call with a "canonical" (correct) one, and lets the student continue.
- Branch-Aware Distillation: The model is trained to predict the "repaired" action via Negative Log-Likelihood (NLL) while receiving dense token-level feedback from the teacher specialists on the subsequent reasoning steps.
Datasets
The authors developed Reframe-100K, a dataset consisting of:
- 63K Image QA pairs and 37K Video QA pairs.
- It is built from heterogeneous spatial and temporal grounding annotations.
- Quality Control: The pipeline (Figure 2) filters out "text shortcuts" (questions answerable without vision) and uses a multi-round auditing process with models like Qwen and GPT-4 to ensure the evidence provided is both necessary and sufficient.
Key Insights and Ideas
- Alignment is a State of Mind (and Sight): Traditional alignment focuses on the mapping from $X \rightarrow Y$. Reframe argues that for agents, we must align the intermediate state $S$ (the observation) to ensure the path to the answer is grounded in reality.
- Dropout as a Forcing Function: By taking away the "crutch" of the original image, the model is forced to learn the causal relationship between its tool actions and its reasoning success.
- Repair over Reprimand: Rather than just penalizing a model for a bad tool call, "repairing" the observation mid-rollout allows the model to still learn how to reason, even if it hasn't perfectly mastered the tool-calling part yet.
Key Results
Reframe was evaluated against several strong baselines (Pixel-Reasoner, Thyme, DeepEyes) across 14 benchmarks.
- Superior Performance: Reframe outperformed the strongest external image and video averages by 11.71 and 8.86 points, respectively.
- High Observation Dependence: On the $V^*$ benchmark, Reframe achieved a "dependence score" of 75.53, compared to a maximum of 41.93 for other models. This indicates that Reframe is significantly more sensitive to the content of what its tools return.
- Tool Gain: Enabling tools improved Reframe's performance by 5.23 points, whereas some existing models actually saw a decrease in performance when tools were enabled (suggesting their tool use was more distracting than helpful).
Limitations and Future Work
- Toolset Simplicity: The current framework only uses two fundamental tools (
CropImageandSelectFrames). Expanding this to more complex tools (web search, code execution) remains an open challenge. - Computational Cost: On-policy distillation with multiple teachers and mid-rollout interventions is computationally intensive compared to standard SFT.
- Fixed Teachers: The specialists are trained first and then held fixed. A co-evolutionary approach where teachers and students improve together could be a promising future direction.
For more details, check out the project page.
Summary updated