TL;DR
AnchorUp is a lightweight feature upsampler for Vision Foundation Models (VFMs) that uses classical interpolation as a baseline and only learns to predict the missing high-frequency details. By "anchoring" the model to a parameter-free interpolant, it achieves state-of-the-art accuracy with 75% fewer parameters and 1.6x faster speeds than the strongest existing baselines.
What Problem Does This Paper Address?
Vision Foundation Models like DINOv3 or CLIP are incredibly powerful for understanding images, but they have a "resolution problem." They typically process images in patches (e.g., 14x14 or 16x16 pixels), meaning their output feature maps are much smaller than the original input. For dense tasks like semantic segmentation or object tracking, we need these features at full pixel resolution.
Current "learned upsamplers" try to solve this by training a neural network to regress the entire high-resolution feature map from the low-resolution one. The authors of AnchorUp noticed a major redundancy: most of a feature map consists of smooth regions that simple, "dumb" algorithms like bilinear interpolation already handle perfectly. By forcing a complex network to re-learn these smooth areas, researchers are wasting computational budget and model capacity.
Methods
The core idea of AnchorUp is to stop predicting the whole feature map and start predicting only the correction needed to fix a classical interpolation.
The architecture follows a simple gated formula: $$\hat{F}_{HR} = F_{anchor} + \alpha(F_{learn} - F_{anchor})$$
- The Anchor ($F_{anchor}$): A parameter-free classical upsampler (bilinear or bicubic). It provides the "smooth" base of the features.
- The Guidance Network ($F_{learn}$): A small NAF-style (Neighborhood Attention Filtering) network that looks at the input image to predict what the features should look like.
- The Scalar Gate ($\alpha$): A single learnable value that determines how much the model should trust the learned correction versus the classical anchor.
By framing the problem this way, the guidance network doesn't have to worry about the "easy" parts of the image. It can spend its entire parameter budget (only ~167k parameters) focusing on the "hard" parts: edges, whiskers, and fine textures.
Datasets
The model is trained once on ImageNet using features from a DINOv3-B/16 backbone. To prove its "zero-shot" generalization capabilities, it is then evaluated—without any further training—on several other backbones and benchmarks:
- Semantic Segmentation: Pascal VOC and Cityscapes.
- Video Object Segmentation: DAVIS-2017.
- Backbones tested: DINOv3, Franca, CLIP, SigLIP 2, PEcore, PEspatial, and RADIO.
Key Insights and Ideas
- The "Aha" Moment: Classical interpolation is a surprisingly strong baseline. Learned models often fail to beat "dumb" bicubic interpolation on average because they struggle to maintain the smooth consistency that interpolation provides for free.
- High-Pass Learning: An analysis of the learned correction confirms it acts as a high-pass filter. It stays "flat" (zero) over backgrounds and interiors but "fires" at object boundaries and fine repeated structures.
- Efficiency through Framing: Because the anchor handles the low-frequency content, the guidance network can be much smaller. AnchorUp uses a guidance dimension of $d=128$, whereas the previous state-of-the-art (NAF) required $d=256$.
Key Results
- Speed and Size: AnchorUp is 1.6x faster than NAF and uses only 1/4 of the parameters.
- Segmentation Accuracy: On Cityscapes, AnchorUp (bilinear) achieves a mean IoU of 60.77%, outperforming NAF (60.61%) despite being significantly lighter.
- Boundary Precision: On the DAVIS-2017 video task, AnchorUp shows its greatest strength in contour accuracy ($F$), outperforming NAF across every single backbone tested. This confirms that the model is indeed better at capturing sharp edges.
- Spectral Evidence: Fourier analysis shows that at low frequencies, the correction carries 100x less power than the anchor, proving the network successfully offloads the "easy" work to the classical algorithm.
Limitations and Future Work
- The Global Gate: Currently, $\alpha$ is a single scalar for the entire image. The authors suggest that a spatially-varying gate (allowing the model to trust the anchor more in flat regions and the network more at edges) could be a promising next step.
- Domain Specificity: The study focused on natural images. The authors hypothesize the "anchoring" premise might be even stronger in domains like remote sensing, where images have very distinct flat and structured regions, but this remains to be tested.
- Anchor Choice: The "best" anchor (bilinear vs. bicubic) seems to depend on the specific benchmark, suggesting there isn't a one-size-fits-all classical baseline yet.
Summary updated