AnchorUp: Anchoring Feature Upsampling on a Classical Interpolation

Kotaro Horikawa, Keisuke Toida, Junya Komori, Kazuhiro HottaarXiv 2026 · arXiv

TL;DR

AnchorUp is a lightweight feature upsampler for Vision Foundation Models (VFMs) that uses classical interpolation as a baseline and only learns to predict the missing high-frequency details. By "anchoring" the model to a parameter-free interpolant, it achieves state-of-the-art accuracy with 75% fewer parameters and 1.6x faster speeds than the strongest existing baselines.

What Problem Does This Paper Address?

Vision Foundation Models like DINOv3 or CLIP are incredibly powerful for understanding images, but they have a "resolution problem." They typically process images in patches (e.g., 14x14 or 16x16 pixels), meaning their output feature maps are much smaller than the original input. For dense tasks like semantic segmentation or object tracking, we need these features at full pixel resolution.

Current "learned upsamplers" try to solve this by training a neural network to regress the entire high-resolution feature map from the low-resolution one. The authors of AnchorUp noticed a major redundancy: most of a feature map consists of smooth regions that simple, "dumb" algorithms like bilinear interpolation already handle perfectly. By forcing a complex network to re-learn these smooth areas, researchers are wasting computational budget and model capacity.

Methods

The core idea of AnchorUp is to stop predicting the whole feature map and start predicting only the correction needed to fix a classical interpolation.

The architecture follows a simple gated formula: $$\hat{F}_{HR} = F_{anchor} + \alpha(F_{learn} - F_{anchor})$$

  1. The Anchor ($F_{anchor}$): A parameter-free classical upsampler (bilinear or bicubic). It provides the "smooth" base of the features.
  2. The Guidance Network ($F_{learn}$): A small NAF-style (Neighborhood Attention Filtering) network that looks at the input image to predict what the features should look like.
  3. The Scalar Gate ($\alpha$): A single learnable value that determines how much the model should trust the learned correction versus the classical anchor.
Figure 2. AnchorUp. (a) The low-resolution feature map is upsampled in two ways: by a classical upsampler with no parameters, which gives the anchor, and by a NAF-style upsampler, a guidance network followed by neighborhood cross-attention. In the attention, queries come from the encoded image, keys from its average pooling, and values from the feature map itself. Only the guidance network and the gate α = σ(γ) have learnable parameters; the gate controls how far the output moves from the anchor toward the NAF-style prediction. (b) The output is a point on the segment between the two predictions: the network picks the direction, the gate picks the distance. The dashed line is the remaining error to the target.

By framing the problem this way, the guidance network doesn't have to worry about the "easy" parts of the image. It can spend its entire parameter budget (only ~167k parameters) focusing on the "hard" parts: edges, whiskers, and fine textures.

Datasets

The model is trained once on ImageNet using features from a DINOv3-B/16 backbone. To prove its "zero-shot" generalization capabilities, it is then evaluated—without any further training—on several other backbones and benchmarks:

  • Semantic Segmentation: Pascal VOC and Cityscapes.
  • Video Object Segmentation: DAVIS-2017.
  • Backbones tested: DINOv3, Franca, CLIP, SigLIP 2, PEcore, PEspatial, and RADIO.

Key Insights and Ideas

  • The "Aha" Moment: Classical interpolation is a surprisingly strong baseline. Learned models often fail to beat "dumb" bicubic interpolation on average because they struggle to maintain the smooth consistency that interpolation provides for free.
  • High-Pass Learning: An analysis of the learned correction confirms it acts as a high-pass filter. It stays "flat" (zero) over backgrounds and interiors but "fires" at object boundaries and fine repeated structures.
  • Efficiency through Framing: Because the anchor handles the low-frequency content, the guidance network can be much smaller. AnchorUp uses a guidance dimension of $d=128$, whereas the previous state-of-the-art (NAF) required $d=256$.
Visualizing the decomposition: the anchor provides the shape, while the correction adds the fine details like the tiger's whiskers.

Key Results

  • Speed and Size: AnchorUp is 1.6x faster than NAF and uses only 1/4 of the parameters.
  • Segmentation Accuracy: On Cityscapes, AnchorUp (bilinear) achieves a mean IoU of 60.77%, outperforming NAF (60.61%) despite being significantly lighter.
  • Boundary Precision: On the DAVIS-2017 video task, AnchorUp shows its greatest strength in contour accuracy ($F$), outperforming NAF across every single backbone tested. This confirms that the model is indeed better at capturing sharp edges.
  • Spectral Evidence: Fourier analysis shows that at low frequencies, the correction carries 100x less power than the anchor, proving the network successfully offloads the "easy" work to the classical algorithm.
Fourier power spectrum showing the correction focusing on high-frequency details.

Limitations and Future Work

  • The Global Gate: Currently, $\alpha$ is a single scalar for the entire image. The authors suggest that a spatially-varying gate (allowing the model to trust the anchor more in flat regions and the network more at edges) could be a promising next step.
  • Domain Specificity: The study focused on natural images. The authors hypothesize the "anchoring" premise might be even stronger in domains like remote sensing, where images have very distinct flat and structured regions, but this remains to be tested.
  • Anchor Choice: The "best" anchor (bilinear vs. bicubic) seems to depend on the specific benchmark, suggesting there isn't a one-size-fits-all classical baseline yet.

Summary updated

Abstract

Vision foundation models produce semantically strong features on a coarse patch grid, and dense prediction needs them at pixel resolution. Existing learned upsamplers close that gap by regressing the high-resolution feature map from the low-resolution one, typically through cross-attention driven by the input image. We observe that most of what they regress needs no learning: plain bilinear interpolation already reproduces the smooth regions of the target, and recent benchmarks even report bilinear and bicubic on par with several learned upsamplers. The rest is mostly edges and fine details, which suggests that these learned upsamplers spend much of their capacity re-computing the smooth content that classical interpolation provides. We therefore introduce AnchorUp, which stops predicting the high-resolution feature map and predicts only its correction from the interpolation. It keeps a parameter-free classical upsampler as an anchor, obtains a second prediction from a small guidance network, and blends the two through a single learnable scalar gate. Because the anchor supplies the smooth content, the guidance network only has to correct what remains. An analysis of the learned correction confirms that it concentrates on edges and fine details, and so the network can be small: AnchorUp uses 167,425 parameters, a quarter of the budget of the strongest baseline, NAF, and runs at 64% of its latency. Trained once on DINOv3 with only the first of NAF's two training stages, and evaluated on that backbone plus six held-out ones, it is already on par with NAF trained through both stages on Pascal VOC and Cityscapes. Our method surpasses it on DAVIS-2017 video object segmentation; the gain holds on every backbone and is larger in contour accuracy F. Code is available at https://github.com/KotaroHorikawa/AnchorUp.
BibTeX
@misc{horikawa2026anchorup,
  author = {Kotaro Horikawa and Keisuke Toida and Junya Komori and Kazuhiro Hotta},
  title = {{AnchorUp: Anchoring Feature Upsampling on a Classical Interpolation}},
  year = {2026},
  eprint = {2611.24296},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24296},
}
Figure