They Can See It but Not Say It: Iterative Self Knowledge Re-expression in Visual Reasoning Relative Pose Identification

Mengyu Wang, Ken Deng, Yftah Ziser, Shay B. Cohen, Tiejun MaarXiv 2026 · arXiv

TL;DR

Modern Vision-Language Models (VLMs) possess latent 3D spatial knowledge but suffer from an "expression bottleneck" that causes them to output biased, near-random guesses on camera pose tasks. This paper introduces SKR-ID, an annotation-free pipeline that unearths this hidden knowledge by replacing generic text generation with task-specific classification heads and using an iterative "median-logit" debiasing trick to force the model to learn balanced spatial reasoning.

What Problem Does This Paper Address?

While VLMs like LLaVA or Qwen excel at describing what is in a 2D image, they are notoriously bad at 3D geometry. Specifically, they struggle with Visual Reasoning Relative Pose Identification (VRRPI)—the ability to look at two images of the same scene and determine how the camera moved between them (e.g., "Did the camera move left or right?").

The authors discovered that this isn't just a lack of knowledge; it’s a systematic failure in how models "talk." When faced with VRRPI tasks, frontier VLMs exhibit a "collapsed baseline": they develop a stubborn preference for one answer (like always picking "right") regardless of what the images actually show. This bias persists even if you swap the order of the images or the multiple-choice options.

Figure 1: Example instance of the VRRPI task.

Methods

The authors propose Self Knowledge Re-expression with Iterative Debiasing (SKR-ID). The core idea is that the model "knows" the answer internally but loses it during the process of generating text tokens. SKR-ID fixes this in two stages:

  1. Cold-start Initialization: Instead of asking the model about 3D camera motion directly, they use an "object-centric" prompt. They ask the model to identify a shared object in both frames and describe its 2D movement. Since VLMs are already good at 2D tracking, this anchors the initial reasoning in actual visual evidence, producing a much more balanced set of "pseudo-labels" for training.
  2. Iterative Re-expression & Debiasing:
    • Re-expression: They strip away the standard text-generation head and attach a lightweight classification head to the model's internal representations. This bypasses the linguistic biases of next-token prediction.
  3. Median-Logit Ranking: To prevent the model from falling back into its old biased habits, they use a dynamic threshold. Instead of assuming a 0.5 probability cutoff, they rank all predictions by their confidence (logits) and set the threshold at the median. This forces a perfect 50/50 label distribution during each self-training iteration, effectively "bullying" the model into considering both possibilities equally.

Datasets

  • Training: The method is entirely annotation-free. It uses randomly extracted, unlabeled image pairs from the ScanNet dataset.
  • Evaluation: Performance is measured on VRRPI-Bench, which includes subsets from 7 Scenes and ScanNet, categorized by the difficulty of the viewpoint change (15° to 60°).

Key Insights and Ideas

  • The Expression Bottleneck: The most profound insight is that VLM failures in 3D reasoning are often "output pathologies" rather than a total absence of geometric understanding. The models "see" the change but can't "say" it correctly through standard autoregressive generation.
  • Stable Biases: The authors proved that VLM errors are not stochastic noise. As shown in Table 1, models like LLaVA-OV showed a 100% bias toward specific options in certain configurations, indicating a deep-seated "indexing bias."
  • 2D as a Bridge to 3D: By forcing the model to first perform 2D object grounding (something it is good at), the researchers successfully "cold-started" a reasoning chain that eventually led to better 3D camera-centric inferences.
Table 1: Prediction ratios under controlled perturbations. Each cell reports the ratio of predicted labels xlefty vs. xrighty. In all experimental settings, the ground-truth distribution is balanced (50.0% / 50.0%). Base, Swap Opt., and Swap Img. denote the base inputs, the option-swap inputs, and the image-swap inputs settings respectively.

Key Results

SKR-ID delivered massive improvements across the board without ever seeing a single human-annotated label:

  • LLaVA-OneVision-7B: Macro-F1 score jumped from a dismal 35.0% (worse than random) to 58.0%.
  • Qwen3-VL-8B: Improved from 39.5% to 53.8%.
  • VST-RL-7B: Even this specialized spatial model saw a boost from 45.4% to 53.7%.

The confusion matrices (analyzed in Section 7.4) show a clear evolution: the base model's output starts as a solid block (total collapse), but after SKR-ID, it transforms into a distinct diagonal, signifying that the model is finally making meaningful, evidence-based predictions.

Table 2: Results (Macro-F1 in %) on VRRPI-Bench.

Limitations and Future Work

  • Model Scope: The study focused on three specific open-source backbones. While they represent the state-of-the-art, the behavior of closed-source giants (like GPT-4o) remains a question.
  • Data Quality: The training used raw, noisy video frames. The authors suggest that adding a quality filter (e.g., removing motion-blurred frames) could further boost performance.
  • Task Complexity: Currently, the method handles binary choices (left vs. right). Extending this to continuous pose estimation or more complex 3D scenes is the logical next step.

Summary updated

Abstract

The task of visual reasoning relative pose identification (VRRPI) has exposed critical weaknesses in the multi-view 3D spatial reasoning capabilities of modern vision-language models (VLMs). We present a systematic analysis of VLMs' behavior in VRRPI, uncovering stable output biases where VLMs disproportionately predict specific motion patterns regardless of visual evidence. To mitigate these inherent biases, we propose self knowledge re-expression with iterative debiasing (SKR-ID), an annotation-free pipeline that adapts VLMs for VRRPI using only randomly extracted unannotated image pairs. SKR-ID transitions VLMs' output mechanism from generic next-token prediction to task-optimized classification heads. The pipeline proceeds in two stages: (1) a cold-start initialization using debiasing object-centric prompts to anchor motion inference in 2D visual evidence; and (2) a recursive logit-ranking mechanism that enforces statistical parity by using the median logit value as a dynamic threshold. Experimental evaluation across frontier VLMs on standard benchmarks demonstrates that SKR-ID significantly mitigates systematic biases. Our method improves open-source VLMs' Macro-F1 scores from 35.0-45.4% to 53.7-58.0%, without using any annotated data. These results show that SKR-ID can effectively improve 3D reasoning performance from a severely collapsed baseline. The code is publicly available at https://github.com/astudentuser/Codes_of_SKR-ID.
BibTeX
@misc{wang2026they,
  author = {Mengyu Wang and Ken Deng and Yftah Ziser and Shay B. Cohen and Tiejun Ma},
  title = {{They Can See It but Not Say It: Iterative Self Knowledge Re-expression in Visual Reasoning Relative Pose Identification}},
  year = {2026},
  eprint = {2611.10483},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10483},
}
Figure