OT-BLS: Optimal-Transport-based Backbone Layer Selection for Frozen DINOv3 in Cross-Domain Few-Shot Segmentation

Natsuo Okamoto, Kazuki Kobayashi, Yoshio Kato, Shuhei TarashimaarXiv 2026 · arXiv

TL;DR

OT-BLS introduces a novel layer-selection criterion for training-free cross-domain few-shot segmentation (CD-FSS) using frozen DINOv3 backbones. By leveraging optimal transport to measure direct correspondence between query and support images, it identifies the most effective feature layer for mask prediction, significantly improving accuracy while running up to 14x faster than previous state-of-the-art selection methods.

What Problem Does This Paper Address?

In cross-domain few-shot segmentation (CD-FSS), the goal is to segment objects in a "query" image from a new domain (e.g., satellite or medical imagery) using only a handful of annotated "support" examples. Recently, researchers have found that using frozen vision foundation models like DINOv3 is highly effective because it avoids the costs and overfitting risks associated with fine-tuning on tiny datasets.

However, there is a catch: the "best" features for segmentation aren't always found in the final layer of the neural network. Depending on the specific image and domain, an intermediate layer might yield a much more accurate mask. While previous methods tried to select the best layer using indirect proxies—like how "confident" the prediction is or how well features are separated—these proxies often fail to capture whether the predicted query foreground actually corresponds to the support foreground.

Methods

The authors propose OT-BLS (Optimal-Transport-Based Backbone Layer Selection). Instead of relying on indirect proxies, OT-BLS directly evaluates the agreement between the support labels and the predicted query mask.

The process works in three main steps:

  1. Candidate Generation: For a given query image, the system generates a candidate segmentation mask for each layer in the DINOv3 backbone (specifically layers 7 through 12).
  2. Optimal Transport (OT) Plan: The core innovation is using an OT algorithm to create a "transport plan" ($P^*$). This plan acts as a soft, patch-level mapping between the query image and the support images based on feature similarity. It essentially asks: "Which parts of the query image look most like which parts of the support images?"
  3. Scoring via Label Agreement: For each candidate layer, the system calculates a score based on how much "mass" is transported to the correct regions. If a query patch is predicted as "foreground," the score increases if the OT plan maps that patch to a "foreground" region in the support images. The final score is a weighted balance of foreground and background precision.
Fig. 1: Overview of OT-BLS for the case K = 3.

Datasets

The authors evaluated OT-BLS across eight diverse benchmark datasets to test its "cross-domain" robustness:

  • Natural Images: COCO-20i, FSS-1000.
  • Satellite Imagery: DeepGlobe, SpaceNetv2.
  • Medical Imagery: ISIC (skin lesions), Chest X-ray, Kvasir-SEG (polyps).
  • Underwater Scenes: SUIM.

Key Insights and Ideas

  • Direct vs. Indirect Evaluation: The "aha moment" is that layer selection should be a direct validation step. By using Optimal Transport, the model can check if its predictions "make sense" relative to the provided examples without needing ground truth for the query image.
  • Precision over F1 Score: The authors intentionally use precision rather than the F1 score for their selection criterion. In CD-FSS, the size of an object can vary wildly between domains (e.g., a "car" in a close-up photo vs. a satellite image). An F1 score would unfairly penalize masks that don't match the size of the support object, whereas precision focuses on label agreement.
  • Robustness to Errors: As shown in Figure 2, the OT-BLS score ($s$) effectively penalizes both under-segmentation (which hurts background precision) and over-segmentation (which hurts foreground precision).
Fig. 2: Values of precisionfg, precisionbg, and s for ground- truth, under-segmented, and over-segmented masks.

Key Results

OT-BLS demonstrates a superior balance of accuracy and efficiency:

  • Accuracy: It outperformed the standard "last-layer" baseline in 12 out of 16 tested scenarios (1-shot and 5-shot across 8 datasets). It improved the average mIoU (mean Intersection over Union) from 55.72% to 57.62%.
  • Ranking: It achieved the best average rank among all compared layer-selection strategies.
  • Speed: Perhaps most impressively, it is significantly faster than the previous top-performing criterion ("Support Prediction mIoU"), running 3.7x faster in 1-shot settings and 14.6x faster in 5-shot settings.
TABLE I: 1-shot and 5-shot performance of each method (mIoU, %). The best value among all methods except Oracle, including the last-layer baseline, is shaded light pink. The best value among the selection strategies, excluding Oracle and the last-layer baseline, is bold. Avg. denotes the mean mIoU over all 16 settings. Rank denotes the corresponding average rank over the same selection strategies. COCO-20i values are averaged over four folds. Oracle and the last-layer baseline are excluded from the rank computation.

Limitations and Future Work

  • Hyperparameter Sensitivity: The weighting parameter $\alpha$ (which balances foreground vs. background importance) currently defaults to 0.5. However, the authors found that the optimal $\alpha$ varies by dataset. Future work could involve adaptively selecting this parameter based on the domain.
  • Search Space: The method currently only searches the later layers of the backbone (layers 7-12), based on the assumption that semantic information is concentrated there. Exploring a wider or more granular search space might yield further gains.
  • Foundation Model Dependency: While tested on DINOv3, the performance on other emerging foundation models remains to be explored.

Summary updated

Abstract

Cross-domain few-shot segmentation (CD-FSS) aims to segment a target class in images from a previously unseen domain using only a few annotated support examples. Training-free CD-FSS methods, which typically leverage vision foundation models such as DINOv3, achieve competitive performance without parameter updates. However, the optimal DINOv3 layer for mask prediction varies across support--query episodes. Although several layer-selection criteria have been proposed, they do not explicitly evaluate whether the predicted query foreground and background labels agree with the corresponding support labels. To address this limitation, we propose a novel layer-selection criterion, Optimal-Transport-Based Backbone Layer Selection (OT-BLS). Our key insight is that an effective layer-selection criterion can be derived from the optimal-transport correspondence between query and support patch-token features. Based on this insight, we first estimate the transport plan using a well-established optimal transport algorithm and then compute a layer-selection score by aggregating the transported mass according to the foreground--background partition defined by each candidate query mask. The resulting score reflects how well the predicted foreground and background of the query correspond to those of the support. Experiments on eight benchmark datasets demonstrate the effectiveness of OT-BLS. It outperforms the conventional last-layer baseline in 12 of 16 dataset/shot settings. Furthermore, OT-BLS improves the average mIoU from 55.72 to 57.62 and achieves a 2.99-point improvement over the best-performing existing layer-selection criterion while running 3.7--14.6$\times$ faster.
BibTeX
@misc{okamoto2026ot,
  author = {Natsuo Okamoto and Kazuki Kobayashi and Yoshio Kato and Shuhei Tarashima},
  title = {{OT-BLS: Optimal-Transport-based Backbone Layer Selection for Frozen DINOv3 in Cross-Domain Few-Shot Segmentation}},
  year = {2026},
  eprint = {2611.15187},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15187},
}
Figure