Introduction: The Resolution Bottleneck in VFMs
Vision Foundation Models (VFMs), such as DINOv2, have become essential backbones for diverse computer vision tasks. However, their reliance on "patchification"—dividing images into 16x16 blocks—results in low-resolution feature maps. For dense prediction tasks like segmentation, where pixel-level precision is critical, this 16x downsampling represents a significant information loss. While traditional approaches utilize heavy, task-specific decoders to recover detail, a more efficient alternative is the use of lightweight feature upsamplers. This paper introduces a rigorous benchmarking framework to evaluate these upsamplers using Interactive Segmentation (IS).
The Interactive Segmentation Benchmark
The authors propose Interactive Segmentation as the ideal "stress test" for feature upsampling. IS requires a model to refine object boundaries based on user-provided "clicks." This task demands both high-level semantic understanding to identify the object and fine-grained spatial accuracy to delineate boundaries.
The experimental setup involves a frozen VFM backbone to isolate the performance of the upsampling module. The study evaluates five strategies: a low-resolution baseline, standard bilinear interpolation, and modern image-adaptive learnable upsamplers including LiFT, FeatUp (JBU), and LoftUp.
Architectural Considerations: Click Injection
A key component of the benchmark is determining how to integrate user interaction into the model. The researchers explored two primary "click injection" strategies: Early Injection, adding click information to the image features before they enter the VFM; and Late Injection, integrating click information after the VFM backbone.
The study found that the optimal injection point depends on the complexity of the encoder. Simple encoders benefit from early injection, whereas more complex architectures like SimpleViT perform better with late injection.
Comparative Analysis of Upsampling Strategies
The benchmark revealed significant performance disparities between methods:
- LoftUp Dominance: LoftUp emerged as the superior method, achieving an IoU of 78.49% on the GrabCut dataset, compared to 65.04% for bilinear upsampling. Its success is attributed to its use of full-resolution pseudo-ground truth and cross-attention mechanisms.
- The Bilinear Artifact Trap: Interestingly, training on low-res features and upscaling the final mask outperformed bilinear upscaling of the features themselves. This suggests that simple interpolation introduces artifacts that hinder the segmentation head.
- Efficiency Gains: LoftUp drastically reduced the human effort required, needing only 4.66 clicks to reach 90% IoU on GrabCut, versus 10.58 clicks for the bilinear method.
Conclusion
By standardizing the evaluation of feature upsamplers through Interactive Segmentation, this work demonstrates that the choice of upsampling strategy is as crucial as the backbone itself. LoftUp provides a significant leap in both accuracy and efficiency, offering a path toward high-precision dense prediction without the need for computationally expensive decoders. Future research will focus on better integrating these modules into multiscale architectures to further enhance boundary precision.
Summary updated