Benchmarking Feature Upsampling Methods for Vision Foundation Models using Interactive Segmentation

Volodymyr Havrylov, Haiwen Huang, Dan Zhang, Andreas GeigerarXiv 2025 · DOI · arXiv

Introduction: The Resolution Bottleneck in VFMs

Vision Foundation Models (VFMs), such as DINOv2, have become essential backbones for diverse computer vision tasks. However, their reliance on "patchification"—dividing images into 16x16 blocks—results in low-resolution feature maps. For dense prediction tasks like segmentation, where pixel-level precision is critical, this 16x downsampling represents a significant information loss. While traditional approaches utilize heavy, task-specific decoders to recover detail, a more efficient alternative is the use of lightweight feature upsamplers. This paper introduces a rigorous benchmarking framework to evaluate these upsamplers using Interactive Segmentation (IS).

The Interactive Segmentation Benchmark

The authors propose Interactive Segmentation as the ideal "stress test" for feature upsampling. IS requires a model to refine object boundaries based on user-provided "clicks." This task demands both high-level semantic understanding to identify the object and fine-grained spatial accuracy to delineate boundaries.

The Interactive Segmentation (IS) workflow: user clicks and previous masks are processed by the IS model to refine the object mask.

The experimental setup involves a frozen VFM backbone to isolate the performance of the upsampling module. The study evaluates five strategies: a low-resolution baseline, standard bilinear interpolation, and modern image-adaptive learnable upsamplers including LiFT, FeatUp (JBU), and LoftUp.

Architectural Considerations: Click Injection

A key component of the benchmark is determining how to integrate user interaction into the model. The researchers explored two primary "click injection" strategies: Early Injection, adding click information to the image features before they enter the VFM; and Late Injection, integrating click information after the VFM backbone.

Comparison of the common IS architecture (a) with the proposed benchmark architectures using Early (b) and Late (c) click injection.

The study found that the optimal injection point depends on the complexity of the encoder. Simple encoders benefit from early injection, whereas more complex architectures like SimpleViT perform better with late injection.

Comparative Analysis of Upsampling Strategies

The benchmark revealed significant performance disparities between methods:

  • LoftUp Dominance: LoftUp emerged as the superior method, achieving an IoU of 78.49% on the GrabCut dataset, compared to 65.04% for bilinear upsampling. Its success is attributed to its use of full-resolution pseudo-ground truth and cross-attention mechanisms.
  • The Bilinear Artifact Trap: Interestingly, training on low-res features and upscaling the final mask outperformed bilinear upscaling of the features themselves. This suggests that simple interpolation introduces artifacts that hinder the segmentation head.
  • Efficiency Gains: LoftUp drastically reduced the human effort required, needing only 4.66 clicks to reach 90% IoU on GrabCut, versus 10.58 clicks for the bilinear method.

Conclusion

By standardizing the evaluation of feature upsamplers through Interactive Segmentation, this work demonstrates that the choice of upsampling strategy is as crucial as the backbone itself. LoftUp provides a significant leap in both accuracy and efficiency, offering a path toward high-precision dense prediction without the need for computationally expensive decoders. Future research will focus on better integrating these modules into multiscale architectures to further enhance boundary precision.

Summary updated

Abstract

Vision Foundation Models (VFMs) are large-scale, pre-trained models that serve as general-purpose backbones for various computer vision tasks. As VFMs' popularity grows, there is an increasing interest in understanding their effectiveness for dense prediction tasks. However, VFMs typically produce low-resolution features, limiting their direct applicability in this context. One way to tackle this limitation is by employing a task-agnostic feature upsampling module that refines VFM features resolution. To assess the effectiveness of this approach, we investigate Interactive Segmentation (IS) as a novel benchmark for evaluating feature upsampling methods on VFMs. Due to its inherent multimodal input, consisting of an image and a set of user-defined clicks, as well as its dense mask output, IS creates a challenging environment that demands comprehensive visual scene understanding. Our benchmarking experiments show that selecting appropriate upsampling strategies significantly improves VFM features quality. The code is released at https://github.com/havrylovv/iSegProbe
BibTeX
@misc{havrylov2025benchmarking,
  author = {Volodymyr Havrylov and Haiwen Huang and Dan Zhang and Andreas Geiger},
  title = {{Benchmarking Feature Upsampling Methods for Vision Foundation Models using Interactive Segmentation}},
  year = {2025},
  eprint = {2505.02075},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.02075},
}
Figure