Faster or Stronger: Towards Flexible Visual Place Recognition via Weighted Aggregation and Token Pruning

Zichao Zeng, June Moh Goo, Junwei Zheng, Weijia Fan, Jiaming Zhang, Rainer Stiefelhagen, Jan BoehmarXiv 2026 · arXiv

TL;DR

This paper introduces a unified framework to make Vision Transformer (ViT)-based Visual Place Recognition (VPR) both more accurate and computationally flexible. By weighting feature clusters based on their importance (WeiAD) and distilling this importance to prune redundant tokens early in the network (WeiToP), the authors achieve state-of-the-art accuracy while allowing on-demand control over the accuracy-efficiency trade-off at inference time without any retraining.

What Problem Does This Paper Address?

Visual Place Recognition (VPR) is the task of matching a query image to a massive database of reference images to determine its location—a core component of autonomous driving and robotics. While modern Vision Transformers (ViTs) like DINOv2 have drastically improved VPR robustness against seasonal and illumination changes, they suffer from two major bottlenecks:

  1. Uniform Aggregation: Standard methods treat all local feature clusters equally during global descriptor aggregation. In reality, a cluster capturing a stable building contributes much more to place recognition than a cluster capturing the sky or a road.
  2. Computational Latency: ViTs are computationally heavy. Most efficiency research focuses on compressing the final global descriptor, completely ignoring the massive computational cost of the feature extraction backbone itself.

The authors set out to answer a compelling question: Can we achieve a flexible balance between efficiency and accuracy by reducing local feature redundancy without sacrificing performance ceilings?

Methods

The authors propose a two-pronged solution: WeiAD for smarter feature aggregation and WeiToP for early token pruning.

Figure 3. The unified framework of WeiAD and WeiToP. At training stage, we fine-tune the late layers of DINOv2 ViT-B on GSV-Cities, alongside the score projection, dimension reduction module, WeiToP module, and weight parameters (fire icon).

1. Weighted Aggregated Descriptor (WeiAD)

Building on Optimal Transport (OT) aggregation baselines like SALAD, WeiAD formulates token-to-cluster assignment as an entropy-regularized OT problem. Crucially, it introduces:

  • Bidirectional Dustbins: A dustbin cluster to absorb uninformative tokens and a ghost token to handle empty clusters.
  • Tiered Weighting Scheme: Clusters are ranked by their raw contribution scores and partitioned into tiers. Highly informative clusters receive larger weights, while marginally relevant ones are suppressed, leading to a more discriminative global descriptor.

2. Token Pruning Module (WeiToP)

To accelerate feature extraction, the authors introduce a self-distillation mechanism. The final aggregation stage acts as a "teacher," explicitly knowing which tokens contributed most to the global descriptor. This token-level importance is distilled into a lightweight "student" MLP attached to an early transformer layer.

At inference time, WeiToP predicts token importance scores, combines them with token embedding norms, and drops the least informative tokens (e.g., sky, road) according to a user-defined retention ratio ($\rho$).

Datasets

The model is trained on the GSV-Cities dataset. Evaluation is performed across a diverse suite of standard VPR benchmarks:

  • MSLS (validation and challenge sets): Large-scale urban scenes with long-term variations.
  • Nordland: Perfectly aligned railway videos captured across four seasons.
  • Pitts250k-test: Google Street View images with significant viewpoint changes.
  • SPED-test: Surveillance camera footage capturing extreme appearance changes.
  • AmsterTime: Historical archival images matched with modern street-view imagery.

Key Insights and Ideas

  • The Aggregation Plan is a Free Supervisor: Instead of relying on complex external labels or heuristics for token pruning, the network uses its own final optimal transport assignment mass as a soft supervision signal.
  • Train Once, Prune Dynamically: Unlike generic pruning methods that require retraining for specific compression targets, WeiToP allows users to dynamically adjust the token retention ratio ($\rho$) on the fly at inference time.
  • Pruning Can Maintain (or Improve) the Performance Ceiling: Dropping uninformative tokens early doesn't just save time; at high retention ratios, removing "harmful" tokens (like dynamic objects or uniform sky) can actually improve retrieval accuracy.

Key Results

WeiAD sets a new state-of-the-art benchmark across all evaluated datasets. When paired with WeiToP, it provides an unprecedented accuracy-efficiency trade-off curve.

Figure 1. Star-shaped markers correspond to WeiAD-based mod- els. The solid red star denotes base WeiAD. The yellow line shows our VPR-specific token pruning approach WeiToP inte- grated with WeiAD across different retention ratios. Other lines indicate WeiAD equipped with different generic token pruning strategies. Single markers show competing VPR methods.
  • Outperforming CNN Baselines: At a token retention ratio of $\rho = 0.5$, WeiAD+WeiToP runs faster than MixVPR while delivering a 3.1% higher Recall@1. At $\rho = 0.4$, it becomes 0.29 ms faster per image than the fastest CNN baseline (Conv-AP) while boosting average Recall@1 by 5.8%.
  • Superiority Over Generic Pruning: WeiToP consistently outperforms general vision token-pruning methods (such as ToMe, FastV, and DynamicViT), which suffer from severe performance drops when forced into aggressive pruning regimes.

Limitations and Future Work

While highly effective, the authors note a few areas for improvement:

  • Fixed Hyperparameters: The cluster structure and tier partitions are currently fixed hyperparameters determined during training rather than being learned adaptively.
  • Early Layer Semantic Blindness: Because token importance is estimated at an early layer to maximize speedups, the model might miss high-level semantic context needed for extremely challenging viewpoint changes.
  • Architecture Constraints: The current formulation is strictly tailored for transformer-based backbones and cannot be directly applied to pure CNN architectures.

Summary updated

Abstract

Visual Place Recognition (VPR) aims to match a query image to reference images of the same place in a large-scale database. Recent state-of-the-art methods employ Vision Transformers (ViTs) as backbone foundation models to extract patch-level features that are robust to viewpoint, illumination, and seasonal variations, which are then aggregated into a compact global descriptor for retrieval. Most existing aggregation methods uniformly pool patch tokens into learned clusters, despite the fact that different clusters often encode distinct spatial or semantic patterns and contribute unequally to VPR performance. To address this limitation, we propose Weighted Aggregated Descriptor (WeiAD), which assigns weights to clusters during aggregation, producing more discriminative global representations. Beyond accuracy, retrieval latency is a critical concern for large-scale deployments and resource-constrained edge devices. Prior work mainly reduces latency by compressing global descriptors, while overlooking the cost of feature extraction, an issue exacerbated by ViT-based backbones. We therefore introduce WeiToP, a VPR-oriented token pruning framework that reduces feature extraction cost via self-distillation, where aggregation-induced token importance supervises a lightweight pruning module attached to an early transformer layer, enabling inference-time token pruning. After a single joint training phase, WeiToP enables plug-and-play token pruning at inference time, allowing flexible and on-demand control over the accuracy-efficiency trade-off without additional training. Moreover, WeiToP outperforms existing token pruning methods adapted from general vision tasks.
BibTeX
@misc{zeng2026faster,
  author = {Zichao Zeng and June Moh Goo and Junwei Zheng and Weijia Fan and Jiaming Zhang and Rainer Stiefelhagen and Jan Boehm},
  title = {{Faster or Stronger: Towards Flexible Visual Place Recognition via Weighted Aggregation and Token Pruning}},
  year = {2026},
  eprint = {2605.20551},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.20551},
}

Loading related content...

Figure