Teaching Vision-Language Models Depth Order with a Dense Soft-Rank Loss

Vision-language models describe what is in an image well and often fail at where things are. They mix up "closer" and "farther," mis-order objects, and give poor distance estimates. We propose to fix part of this with a cheap training signal. A small side branch on the vision encoder is trained to put every image patch in the correct near-to-far order. The loss is a differentiable version of a rank correlation, computed over thousands of patches at once. The language model receives this branch's output added onto its usual visual tokens, so no tokens are added and no 3D reconstruction model runs at inference.

The bet has three parts. First, ordinal depth (which patch is nearer) is easier to learn and more transferable than metric depth, because it does not depend on camera calibration. Second, keeping this "where" branch separate from the semantic "what" features avoids the interference that shared heads are known to suffer. Third, the newer sorting relaxations make a dense listwise rank loss affordable at vision-transformer token counts, where older relaxations run out of memory. If this works, a 3B open model should close much of the gap to systems that need explicit point clouds. Its spatial answers should also depend less on object identity.

Core Claim

Supervising a decoupled per-patch geometric channel with a dense, scale-free ranking objective improves the ordinal and relative-position spatial reasoning of a VLM more than three alternatives at equal data and compute:

  • metric or scale-invariant depth regression on the same channel;
  • sampled pairwise ranking losses;
  • the same loss applied to shared, non-decoupled features.

The gain should hold up when benchmark answers are reweighted to remove dataset shortcuts. The claim is falsifiable at each of those three contrasts. The proposal is designed so that a tie with scale-invariant regression would be reported as the finding, not argued away.

Where this sits

Several papers motivate the gap.

  • What's "up" with vision-language models? shows that spatial prepositions appear in under 0.2% of web captions. It also shows that fine-tuning on spatial data tends to teach dataset priors rather than geometry. That warns against our method becoming one more source of priors.
  • Vision Language Models See What You Want but not What You See finds that scale alone does not fix perspective-taking. This suggests the missing ingredient is the training signal, not capacity.
  • Hidden Sensitivity in Spatial Reasoning Evaluation shows that VSI-Bench rankings are distorted by object category, video source and answer distributions. It corrects this only after the fact, through factor-normalized scoring. We use their Factor Sensitivity Test as an evaluation lens on whether a training-time intervention reduces that dependence.

The closest method is GEODE, from Beyond Flatlands. It decouples spatial reasoning from text generation in two ways. A cross-attention module fuses point clouds and camera poses into special tokens, and an MLP head regresses numbers from a hidden state. We differ in three respects:

  • We need no external 3D pipeline at inference.
  • Our supervision is ordinal rather than L1/L2 metric regression.
  • Our geometric signal is dense over patches rather than read from one regression token.

SpatialThinker takes the reinforcement-learning route, using GRPO with format, count, accuracy and box-overlap rewards. It needed lexicographic gating to stop reward hacking. We replace sampled rewards with an exact gradient. ViewSpatial-Bench fine-tunes a 3B model on 18 discretized directions. That works on its own benchmark, but it is exactly the kind of label-space fine-tuning Kamath et al. warn may not transfer.

The decoupling design borrows from three sources.

  • PSOL showed that separating class-agnostic localization from classification removes the bias toward the most discriminative object parts. It gained over 10% localization accuracy on CUB and ImageNet.
  • The asymmetric decoupled head of Lu et al. showed similar gains for detection.
  • Urrutia et al. prove that a single attention head cannot be both strongly positional and strongly content-driven. This is a theoretical reason to expect geometry and semantics to compete when forced into shared features.

These works are all 2D, single-object, or mechanistic. None trains a geometric channel for a VLM.

The loss itself comes from the sorting literature.

That paper is what makes a dense loss feasible. These methods were tested only on synthetic sorting, CIFAR and tabular learning-to-rank, so applying them to dense geometric supervision is new territory.

The attention-cost direction enters as a hard design constraint, not as the main contribution. GEODE's extra cross-attention, and any scheme that adds geometric tokens, lengthens the sequence the language model must attend over. We add geometric information by summation onto existing visual tokens. The language model's attention cost is therefore unchanged, and the only new cost is a training-time loss.

Research Program

We start from Qwen2.5-VL-3B, chosen because reported numbers exist for it on ViewSpatial-Bench. We attach a geometric branch to the output of its vision encoder: a few lightweight transformer or convolutional layers on a detached copy of the patch features. For each patch $i$, the branch produces a scalar depth score $g_i$ and a geometric embedding $e_i$. A zero-initialized linear projection adds $e_i$ to that patch's visual token before it enters the language model.

The training loss on the branch is one minus the Spearman correlation between soft ranks of the predicted scores and true ranks of ground-truth depth:

$$\mathcal{L}_{\text{rank}} = 1 - \operatorname{corr}\big(r_\epsilon(g),\ \operatorname{rank}(d)\big)$$

The symbols are as follows:

  • $g = (g_1,\dots,g_n)$ are the branch's scores over the $n$ patches of a frame.
  • $d$ is the ground-truth depth averaged within each patch.
  • $\operatorname{rank}(d)$ is the ordinary rank of each patch's depth.
  • $r_\epsilon$ is the soft-rank operator of Blondel et al., where $\epsilon$ sets how close it is to hard ranking.

The loss is invariant to any monotone rescaling of depth, so camera intrinsics and metric scale never enter. Patches with invalid depth are dropped from the ranking.

Ground-truth depth comes from RGB-D indoor scans such as ScanNet, the scan source used by ViewSpatial-Bench. We exclude any scene appearing in our evaluation sets. Training then has two stages:

  1. Branch pre-training. The branch alone is trained on frames with the rank loss.
  2. Joint fine-tuning. The model is fine-tuned on a generic, non-spatial instruction mixture plus the rank loss. Spatial QA is deliberately left out, so any spatial gain must come from the geometric channel rather than from benchmark-style questions.

One engineering piece is required. A frame has roughly one to four thousand patches. PAV, the pool-adjacent-violators algorithm that solves the isotonic regression, is usually implemented sequentially on CPU. We will write a batched GPU version. The fallback is to compute soft ranks within overlapping spatial windows plus a coarse global ranking of window means. We will report wall-clock time and memory against SoftSort at the same $n$, to document where the quadratic method breaks.

Baselines

The naive reference point is the unmodified Qwen2.5-VL-3B. It is also fine-tuned on the same generic instruction mixture for the same steps without the branch, which isolates the effect of the extra fine-tuning. ViewSpatial-Bench reports 35.85% for the base model (derived as 82.09% minus the 46.24% gain), and GPT-4o is at 34.98% on that benchmark. Beating this reference by a clear margin on the relative-position and distance dimensions of VSI-Bench is the minimum bar.

The strongest existing method on VSI-Bench is GEODE (Beyond Flatlands). It reports an overall VSI-Bench score of 37.0 at 1.5B parameters, 25.7 on absolute distance, and 61.9 on counting. For comparison, that paper reports 14.8 on absolute distance for Qwen2.5-VL-7B and 34.0 overall for GPT-4o. GEODE uses explicit 3D inputs at inference that our model does not. We will compare against its reported numbers, and we will be explicit that matching it without point clouds is the target, not exceeding it. We will also report the full set of eight VSI-Bench dimensions for our runs. Beating GEODE on ordinal and relative-position dimensions with only RGB input would count as a strong result. Being within a few points overall would count as a meaningful one. We cannot fully equalize compute with a reported number, so for a compute-fair comparison we train a GEODE-style regression-token head on our backbone with our data.

For ViewSpatial-Bench the strongest reported system is MVSM at 82.09%. It was fine-tuned on that benchmark's own label space, so we treat it as an in-distribution ceiling rather than a fair competitor. We report transfer in both directions instead.

Three ablations isolate the key ingredients. All are run on the same backbone, the same frames and depth maps, the same number of optimizer steps, and an equal hyperparameter sweep of eight configurations each.

  • Loss ablation. Keep the decoupled branch but replace the rank loss with a scale-and-shift-invariant per-patch depth regression. This tests whether ranking matters or whether being calibration-free is all that matters.
  • Pairwise ablation. Keep the branch and use a pairwise logistic ranking loss on sampled patch pairs, with the pair count chosen so memory and time match the listwise loss.
  • Decoupling ablation. Remove the branch and apply the rank loss to a linear readout of the shared visual tokens directly, so geometric gradients flow into the semantic features.

For each ablation, beating it means a gap larger than the seed-to-seed spread across three seeds on the VSI-Bench relative-position and distance dimensions.

Why not just…?

Why not just regress scale-invariant depth per patch? This is the most serious threat. Monocular depth work routinely uses scale-and-shift-invariant regression, which is also calibration-free and dense. Our reason to expect ranking to win is narrow. A rank loss weights every ordering mistake equally, regardless of depth magnitude. It also ignores noise in depth values that does not change their order. Both properties fit questions of the form "which is closer," and neither fits metric regression. Still, nothing in the file shows this empirically, and it may tie. The loss ablation settles it. If the two tie on spatial QA, we will report that the useful ingredient is a decoupled, calibration-free geometric channel, and ranking adds nothing beyond that.

Why not just use a pairwise ranking loss, which needs no sorting machinery? With tens of objects, a pairwise loss over all pairs is cheap, and we would not need soft sorting. The dense patch setting changes this. All pairs means millions of terms per frame, and sampling a tractable subset gives a noisy, low-coverage signal. The listwise soft rank uses the full ordering in $O(n)$ memory (Fast Differentiable Sorting and Ranking). The pairwise ablation tests this directly at equal compute. If sampled pairs match the listwise loss, the computational argument for soft sorting fails, and we will say so.

Why not just fine-tune on spatial QA or optimize spatial rewards with reinforcement learning? Kamath et al. (What's "up"; What's "up" with vision-language models? Investigating their struggle with spatial reasoning) report that spatial QA fine-tuning instills linguistic priors that fail on controlled benchmarks. SpatialThinker (SpatialThinker) needed gated multi-objective rewards to avoid reward hacking. Score-function ranking estimators have high variance, as discussed by Oosterhuis and Dadaneh et al.. These methods do work: SpatialThinker reports 65.9% on MM-Star and 76.3% on RoboSpatial-Home. They are complementary rather than rival, since the geometric channel could sit underneath an RL-trained model. Our test is whether, without any spatial QA in training, we transfer better to the controlled What'sUp benchmark and the concept zero-shot split of Visual Spatial Reasoning than a model fine-tuned on spatial QA.

Why not just feed point clouds as GEODE does? When reliable 3D input is available, GEODE is likely better on metric distance. It does, however, depend on an external, frozen reconstruction pipeline and an extra cross-attention module. Our setting is RGB-only at inference with unchanged attention cost. The comparison above states plainly that we aim to approach GEODE without its inputs, not to beat it on its own terms.

Decisive Experiments

VSI-Bench. The primary evaluation is VSI-Bench's eight dimensions, scored both raw and with the factor-normalized FN-VSI scoring of Lee et al. (Hidden Sensitivity). We also run their Factor Sensitivity Test on a held-out pool drawn from VSI-590K. The specific prediction is about decoupling: the decoupled-branch model should show lower mutual information between its answers and object category than both the shared-feature ablation and the base model. Lee et al. report only post-hoc evaluation, so this would be the first evidence that a training-time intervention reduces factor sensitivity.

Controlled preposition benchmarks. What'sUp, COCO-Spatial, GQA-Spatial and the VSR concept zero-shot split test whether gains survive when object identity is held fixed or categories are unseen. Reference points from the file are 56% for fine-tuned BLIP and 99% for humans on What'sUp, and roughly 70% for the best models and 95.4% for humans on VSR.

Patch-level probe. We measure Spearman correlation between branch scores and true depth on held-out scans, to check that the channel learned ordinal geometry rather than something incidental.

General capability guard. MM-Star accuracy guards against the geometric signal degrading general ability.

Efficiency. We report peak training memory and step time for the rank loss at $n$ from 500 to 4000 patches, against SoftSort. We also confirm that inference token count and attention FLOPs in the language model are identical to the base model.

Outcomes that would refute the claim. The idea fails in any of three cases. It fails if the scale-invariant regression ablation matches the rank loss on relative-position dimensions. It fails if the shared-feature ablation matches the decoupled branch on both accuracy and factor sensitivity. It fails if gains on VSI-Bench vanish under FN-VSI normalization, which would mean we taught the model a new shortcut rather than geometry.