Reframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning

Haozhi Yuan, Handong Li, Yifan Zhang, Qianshan WeiarXiv 2026 · arXiv

TL;DR

Reframe is a novel post-training framework designed to fix "observation neglect" in visual agents—where models invoke tools (like image cropping) but ignore the results. By treating the observation state as an object of alignment through a two-stage process of specialized reinforcement learning and "observation-gated" distillation, Reframe produces agents that significantly outperform existing methods across 14 image and video benchmarks.

What Problem Does This Paper Address?

Equipping Vision-Language Models (VLMs) with tools—such as the ability to crop an image for better detail or select specific video frames—is essential for complex reasoning. However, current "agentic" models suffer from two major flaws:

  1. Observation Neglect: A model might call a tool to see a detail better, but its final answer remains unchanged regardless of what the tool returns. It "looks" but doesn't "see."
  2. Supervision Sparsity: In standard Reinforcement Learning (RL), if a model looks at the wrong part of an image, it gets a low reward, but it doesn't learn how to reason about the correct part. Conversely, in standard distillation, if a student model looks at the wrong thing, the teacher's feedback is useless because it's conditioned on a "garbage" observation.

The authors argue that we need to align not just the model's output, but the observation state itself, ensuring the model both acquires valid evidence and relies on it.

Methods

Reframe introduces a two-stage training pipeline that bypasses the need for a task-specific Supervised Fine-Tuning (SFT) "cold start."

Stage 1: Ground-Drop-GRPO (GD-GRPO)

This stage trains modality-specific specialists (one for images, one for videos). It uses two clever tricks to force the model to use its tools:

  • Original-Observation Dropout: During training, after a model successfully calls a tool (e.g., crops an image), the original full-sized image is stochastically removed. This forces the model to rely entirely on the tool-returned observation to answer the question.
  • Evidence-Aware Rewards: Instead of just rewarding the final correct answer, the model is rewarded for the quality of its "grounding" (how well its crop/selection overlaps with known important regions).
Overview of Reframe showing how GD-GRPO and OG-MOPD improve upon vanilla RL and MOPD.

Stage 2: Observation-Gated MOPD (OG-MOPD)

This stage consolidates the specialists into a single unified model using Multi-Teacher On-Policy Distillation.

  • Observation Gating: If the student model makes a poor tool call (e.g., crops a blank area), the framework intervenes mid-rollout, replaces the bad call with a "canonical" (correct) one, and lets the student continue.
  • Branch-Aware Distillation: The model is trained to predict the "repaired" action via Negative Log-Likelihood (NLL) while receiving dense token-level feedback from the teacher specialists on the subsequent reasoning steps.

Datasets

The authors developed Reframe-100K, a dataset consisting of:

  • 63K Image QA pairs and 37K Video QA pairs.
  • It is built from heterogeneous spatial and temporal grounding annotations.
  • Quality Control: The pipeline (Figure 2) filters out "text shortcuts" (questions answerable without vision) and uses a multi-round auditing process with models like Qwen and GPT-4 to ensure the evidence provided is both necessary and sufficient.
The Reframe-100K data construction pipeline.

Key Insights and Ideas

  • Alignment is a State of Mind (and Sight): Traditional alignment focuses on the mapping from $X \rightarrow Y$. Reframe argues that for agents, we must align the intermediate state $S$ (the observation) to ensure the path to the answer is grounded in reality.
  • Dropout as a Forcing Function: By taking away the "crutch" of the original image, the model is forced to learn the causal relationship between its tool actions and its reasoning success.
  • Repair over Reprimand: Rather than just penalizing a model for a bad tool call, "repairing" the observation mid-rollout allows the model to still learn how to reason, even if it hasn't perfectly mastered the tool-calling part yet.

Key Results

Reframe was evaluated against several strong baselines (Pixel-Reasoner, Thyme, DeepEyes) across 14 benchmarks.

  • Superior Performance: Reframe outperformed the strongest external image and video averages by 11.71 and 8.86 points, respectively.
  • High Observation Dependence: On the $V^*$ benchmark, Reframe achieved a "dependence score" of 75.53, compared to a maximum of 41.93 for other models. This indicates that Reframe is significantly more sensitive to the content of what its tools return.
  • Tool Gain: Enabling tools improved Reframe's performance by 5.23 points, whereas some existing models actually saw a decrease in performance when tools were enabled (suggesting their tool use was more distracting than helpful).

Limitations and Future Work

  • Toolset Simplicity: The current framework only uses two fundamental tools (CropImage and SelectFrames). Expanding this to more complex tools (web search, code execution) remains an open challenge.
  • Computational Cost: On-policy distillation with multiple teachers and mid-rollout interventions is computationally intensive compared to standard SFT.
  • Fixed Teachers: The specialists are trained first and then held fixed. A co-evolutionary approach where teachers and students improve together could be a promising future direction.

For more details, check out the project page.

Summary updated

Abstract

Agentic reinforcement learning (RL) and multi-teacher on-policy distillation (MOPD) offer complementary routes to post-training. For visual agents, however, tool use presents two related challenges. Models may overlook useful tool-returned observations, while poorly grounded actions can limit the evidence available for subsequent distillation. We introduce Reframe, a post-training framework that treats the observation state, rather than only the output distribution, as an object of alignment. First, Ground-Drop-GRPO (GD-GRPO) trains spatial and temporal specialists with evidence-aware rewards and stochastic removal of the original visual input, encouraging them to acquire grounded evidence and use the returned observations. Second, Observation-Gated-MOPD (OG-MOPD) replaces deficient tool actions with canonical ones mid-rollout and resumes distillation from the repaired observation. The two stages complement each other, as learning to rely on tool-returned observations can help the policy benefit from observation repair. Neither stage requires a task-specific SFT cold start. Extensive experiments over 14 image and video benchmarks demonstrate that Reframe outperforms both Mix-RL and vanilla MOPD. Ablation results show that GD-GRPO increases observation dependence and tool gain compared with outcome-based GRPO.
BibTeX
@misc{yuan2026reframe,
  author = {Haozhi Yuan and Handong Li and Yifan Zhang and Qianshan Wei},
  title = {{Reframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning}},
  year = {2026},
  eprint = {2611.12886},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12886},
}
Figure