TL;DR
Modern Vision-Language Models (VLMs) possess latent 3D spatial knowledge but suffer from an "expression bottleneck" that causes them to output biased, near-random guesses on camera pose tasks. This paper introduces SKR-ID, an annotation-free pipeline that unearths this hidden knowledge by replacing generic text generation with task-specific classification heads and using an iterative "median-logit" debiasing trick to force the model to learn balanced spatial reasoning.
What Problem Does This Paper Address?
While VLMs like LLaVA or Qwen excel at describing what is in a 2D image, they are notoriously bad at 3D geometry. Specifically, they struggle with Visual Reasoning Relative Pose Identification (VRRPI)—the ability to look at two images of the same scene and determine how the camera moved between them (e.g., "Did the camera move left or right?").
The authors discovered that this isn't just a lack of knowledge; it’s a systematic failure in how models "talk." When faced with VRRPI tasks, frontier VLMs exhibit a "collapsed baseline": they develop a stubborn preference for one answer (like always picking "right") regardless of what the images actually show. This bias persists even if you swap the order of the images or the multiple-choice options.
Methods
The authors propose Self Knowledge Re-expression with Iterative Debiasing (SKR-ID). The core idea is that the model "knows" the answer internally but loses it during the process of generating text tokens. SKR-ID fixes this in two stages:
- Cold-start Initialization: Instead of asking the model about 3D camera motion directly, they use an "object-centric" prompt. They ask the model to identify a shared object in both frames and describe its 2D movement. Since VLMs are already good at 2D tracking, this anchors the initial reasoning in actual visual evidence, producing a much more balanced set of "pseudo-labels" for training.
- Iterative Re-expression & Debiasing:
- Re-expression: They strip away the standard text-generation head and attach a lightweight classification head to the model's internal representations. This bypasses the linguistic biases of next-token prediction.
- Median-Logit Ranking: To prevent the model from falling back into its old biased habits, they use a dynamic threshold. Instead of assuming a 0.5 probability cutoff, they rank all predictions by their confidence (logits) and set the threshold at the median. This forces a perfect 50/50 label distribution during each self-training iteration, effectively "bullying" the model into considering both possibilities equally.
Datasets
- Training: The method is entirely annotation-free. It uses randomly extracted, unlabeled image pairs from the ScanNet dataset.
- Evaluation: Performance is measured on VRRPI-Bench, which includes subsets from 7 Scenes and ScanNet, categorized by the difficulty of the viewpoint change (15° to 60°).
Key Insights and Ideas
- The Expression Bottleneck: The most profound insight is that VLM failures in 3D reasoning are often "output pathologies" rather than a total absence of geometric understanding. The models "see" the change but can't "say" it correctly through standard autoregressive generation.
- Stable Biases: The authors proved that VLM errors are not stochastic noise. As shown in Table 1, models like LLaVA-OV showed a 100% bias toward specific options in certain configurations, indicating a deep-seated "indexing bias."
- 2D as a Bridge to 3D: By forcing the model to first perform 2D object grounding (something it is good at), the researchers successfully "cold-started" a reasoning chain that eventually led to better 3D camera-centric inferences.
Key Results
SKR-ID delivered massive improvements across the board without ever seeing a single human-annotated label:
- LLaVA-OneVision-7B: Macro-F1 score jumped from a dismal 35.0% (worse than random) to 58.0%.
- Qwen3-VL-8B: Improved from 39.5% to 53.8%.
- VST-RL-7B: Even this specialized spatial model saw a boost from 45.4% to 53.7%.
The confusion matrices (analyzed in Section 7.4) show a clear evolution: the base model's output starts as a solid block (total collapse), but after SKR-ID, it transforms into a distinct diagonal, signifying that the model is finally making meaningful, evidence-based predictions.
Limitations and Future Work
- Model Scope: The study focused on three specific open-source backbones. While they represent the state-of-the-art, the behavior of closed-source giants (like GPT-4o) remains a question.
- Data Quality: The training used raw, noisy video frames. The authors suggest that adding a quality filter (e.g., removing motion-blurred frames) could further boost performance.
- Task Complexity: Currently, the method handles binary choices (left vs. right). Extending this to continuous pose estimation or more complex 3D scenes is the logical next step.
Summary updated