Geometry-Aware Test-Time Learning for Quantitative Spatial Reasoning

Gege Zhang, Shuaicheng Niu, Gang Dai, Lei Sun, Shuangping HuangarXiv 2026 · arXiv

TL;DR

TTL-SR (Geometry-Aware Test-Time Learning for Quantitative Spatial Reasoning) is a plug-and-play framework that forces Vision-Language Models (VLMs) to obey fundamental laws of 3D geometry during inference. By generating coupled auxiliary queries (e.g., asking for direct, horizontal, and vertical distances simultaneously) and optimizing model parameters on the fly using only unlabeled test data, it eliminates physically impossible predictions and significantly boosts spatial precision without requiring costly 3D retraining.

What Problem Does This Paper Address?

While modern Vision-Language Models (VLMs) excel at semantic tasks, they are notoriously bad at quantitative spatial reasoning—such as estimating the exact 3D distance or directional relationship between objects from a single 2D image.

The root of the problem lies in the scarcity of high-quality 3D spatial annotations, leaving models to learn geometry implicitly. During standard inference, a VLM evaluates spatial queries in complete isolation. Without global geometric constraints, the model frequently outputs physically absurd and contradictory predictions—for instance, estimating a direct distance between a monitor and a laptop that is shorter than the horizontal distance between them (as shown in Comparison of spatial reasoning paradigms).

Comparison of spatial reasoning paradigms

Furthermore, standard unsupervised Test-Time Adaptation (TTA) methods like entropy minimization fail here. They encourage the model to become highly confident in its discrete token selections, which collapses the fine-grained, continuous feature representations necessary for accurate geometric regression.

Methods

TTL-SR introduces a three-phase pipeline designed to bridge continuous physical geometry with the discrete token output space of autoregressive language models:

Figure 2: Overview of the TTL-SR pipeline. The framework operates in three phases: (1) Spatial Query Augmentation generates geometrically coupled auxiliary queries via predefined templates; (2) Adaptive Geometric Consistency Modeling filters raw nu- merical estimates through reliability triggers to construct structured token-level pseudo-labels; and (3) Test-Time Optimization iteratively updates VLM parameters through a multi-objective loss.
  1. Spatial Query Augmentation: Given a target query (e.g., "What is the direct distance..."), the framework uses template-based transformations to generate a set of geometrically coupled auxiliary queries probing the same scene (e.g., vertical distance, horizontal distance, object height, and width).
  2. Adaptive Geometric Consistency Modeling:
  3. Adaptive Geometric Triggering: To prevent the model from learning from its own hallucinations, a lightweight trigger checks if the raw predictions satisfy basic geometric principles (like the Pythagorean theorem or aspect ratio stability) within a tolerance threshold $\tau$. Test-time optimization is only activated for reliable prediction groups.
  4. Structured Token-level Pseudo-labeling: Validated continuous numerical targets are broken down into structured token sequences (integer part, decimal point, fractional part, and unit). Soft pseudo-labels are applied to numeric tokens to model spatial uncertainty gracefully.
  5. Test-Time Optimization: The framework updates low-rank adapter weights (LoRA) injected into the VLM backbone using a multi-objective loss function:
  6. Geometric Consistency Loss ($\mathcal{L}_{geo}$): Combines Maximum Likelihood Estimation (MLE) positive learning with an Unlikelihood Loss (UL) to actively suppress geometrically invalid outlier tokens.
  7. KL Divergence Loss ($\mathcal{L}_{kl}$): Anchors the updating model to the original pre-trained VLM outputs to prevent representation collapse and catastrophic forgetting.
  8. Restricted Entropy Minimization Loss ($\mathcal{L}_{ent}$): Sharpens the model's predictions strictly within the validated geometric intervals.

Datasets

The authors evaluated TTL-SR across three major benchmarks:

  • Q-Spatial Bench: Consisting of Q-Spatial-ScanNet (grounded in precise RGB-D measurements from indoor scenes) and Q-Spatial++ (diverse real-world images under varied conditions).
  • SpatialRGPT-Bench: A spatial VQA benchmark spanning indoor, outdoor, and simulated scenes with region annotations.
  • SPAR-Bench: Specifically utilizing the object-to-object distance prediction task to evaluate fine-grained metric-level reasoning.

Key Insights and Ideas

  • Confidence $\neq$ Correctness in Regression: Traditional test-time adaptation methods prioritize confidence maximization. However, forcing a model to be confident actually homogenizes the feature space, destroying the fine-grained distinctions needed for continuous spatial regression (as analyzed in Feature space evolution).
  • Geometry as a Gatekeeper: Instead of directly enforcing geometric equations as hard loss constraints—which amplifies noise when predictions are highly inaccurate—TTL-SR uses geometry as a reliability filter to determine when to learn.
  • Unified Optimization Space: Spatial attributes are tightly coupled. Optimizing a single dimension (like direct distance) in isolation degrades performance in other dimensions (like vertical distance). They must be regularized together to preserve the global geometric structure.
Feature space evolution

Key Results

  • Substantial Accuracy Gains: TTL-SR achieved an absolute accuracy boost of 6.47% for Qwen3-VL-4B-Instruct and 9.41% for SpatialRGPT-VILA-1.5-8B on the Q-Spatial-ScanNet dataset.
  • Outperforming Existing TTA: While standard methods like Tent and TLM often degraded VLM performance due to representation collapse, TTL-SR consistently reduced the Mean Absolute Relative Error (MARE) across all distance and size categories.
  • Competitive with Closed-Source Giants: A lightweight 4B open-source model equipped with TTL-SR achieved 63.9% accuracy on direct distance estimation, outperforming the massive commercial Gemini-3-Flash (58.3%).
Figure 4: Qualitative comparison of spatial reasoning between the baseline Qwen3-VL-4B-Instruct and our TTL-SR.

Limitations and Future Work

The current framework relies on predefined, template-based query augmentations to establish geometric relationships, which may limit its flexibility in highly complex or unformatted environments. The authors note that future work will focus on developing more flexible constraint formulations and extending this test-time learning paradigm to a broader range of multi-modal reasoning tasks.

Summary updated

Abstract

Quantitative spatial reasoning in visual-language models (VLMs) aims to infer spatial distances and directional relationships among objects in 3D space from a 2D image and a natural language query. Despite recent progress, VLM spatial reasoning remains brittle under distribution shifts, largely due to the high cost of 3D supervision. As a result, models often produce inconsistent or contradictory predictions when faced with novel object configurations or rephrased spatial queries, revealing a misalignment between learned representations and underlying geometry. To address this, we propose TTL-SR, a geometry-aware Test-Time Learning framework for quantitative Spatial Reasoning that leverages geometric consistency constraints and unlabeled test data to adapt models to target domains. Specifically, TTL-SR augments the input query with geometrically coupled auxiliary queries, filters unreliable predictions via adaptive geometric triggering to construct structured token-level pseudo-labels, and updates model parameters under a geometry-aware multi-objective loss using only test data. Experimental results demonstrate that TTL-SR significantly boosts spatial reasoning performance, yielding 6.47% and 9.41% accuracy gains for Qwen3-VL-4B-Instruct and SpatialRGPT-VILA-1.5-8B on Q-Spatial-ScanNet dataset, respectively.
BibTeX
@misc{zhang2026geometry,
  author = {Gege Zhang and Shuaicheng Niu and Gang Dai and Lei Sun and Shuangping Huang},
  title = {{Geometry-Aware Test-Time Learning for Quantitative Spatial Reasoning}},
  year = {2026},
  eprint = {2609.06004},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.06004},
}
Figure