TL;DR
This paper introduces a unified framework to make Vision Transformer (ViT)-based Visual Place Recognition (VPR) both more accurate and computationally flexible. By weighting feature clusters based on their importance (WeiAD) and distilling this importance to prune redundant tokens early in the network (WeiToP), the authors achieve state-of-the-art accuracy while allowing on-demand control over the accuracy-efficiency trade-off at inference time without any retraining.
What Problem Does This Paper Address?
Visual Place Recognition (VPR) is the task of matching a query image to a massive database of reference images to determine its location—a core component of autonomous driving and robotics. While modern Vision Transformers (ViTs) like DINOv2 have drastically improved VPR robustness against seasonal and illumination changes, they suffer from two major bottlenecks:
- Uniform Aggregation: Standard methods treat all local feature clusters equally during global descriptor aggregation. In reality, a cluster capturing a stable building contributes much more to place recognition than a cluster capturing the sky or a road.
- Computational Latency: ViTs are computationally heavy. Most efficiency research focuses on compressing the final global descriptor, completely ignoring the massive computational cost of the feature extraction backbone itself.
The authors set out to answer a compelling question: Can we achieve a flexible balance between efficiency and accuracy by reducing local feature redundancy without sacrificing performance ceilings?
Methods
The authors propose a two-pronged solution: WeiAD for smarter feature aggregation and WeiToP for early token pruning.
1. Weighted Aggregated Descriptor (WeiAD)
Building on Optimal Transport (OT) aggregation baselines like SALAD, WeiAD formulates token-to-cluster assignment as an entropy-regularized OT problem. Crucially, it introduces:
- Bidirectional Dustbins: A dustbin cluster to absorb uninformative tokens and a ghost token to handle empty clusters.
- Tiered Weighting Scheme: Clusters are ranked by their raw contribution scores and partitioned into tiers. Highly informative clusters receive larger weights, while marginally relevant ones are suppressed, leading to a more discriminative global descriptor.
2. Token Pruning Module (WeiToP)
To accelerate feature extraction, the authors introduce a self-distillation mechanism. The final aggregation stage acts as a "teacher," explicitly knowing which tokens contributed most to the global descriptor. This token-level importance is distilled into a lightweight "student" MLP attached to an early transformer layer.
At inference time, WeiToP predicts token importance scores, combines them with token embedding norms, and drops the least informative tokens (e.g., sky, road) according to a user-defined retention ratio ($\rho$).
Datasets
The model is trained on the GSV-Cities dataset. Evaluation is performed across a diverse suite of standard VPR benchmarks:
- MSLS (validation and challenge sets): Large-scale urban scenes with long-term variations.
- Nordland: Perfectly aligned railway videos captured across four seasons.
- Pitts250k-test: Google Street View images with significant viewpoint changes.
- SPED-test: Surveillance camera footage capturing extreme appearance changes.
- AmsterTime: Historical archival images matched with modern street-view imagery.
Key Insights and Ideas
- The Aggregation Plan is a Free Supervisor: Instead of relying on complex external labels or heuristics for token pruning, the network uses its own final optimal transport assignment mass as a soft supervision signal.
- Train Once, Prune Dynamically: Unlike generic pruning methods that require retraining for specific compression targets, WeiToP allows users to dynamically adjust the token retention ratio ($\rho$) on the fly at inference time.
- Pruning Can Maintain (or Improve) the Performance Ceiling: Dropping uninformative tokens early doesn't just save time; at high retention ratios, removing "harmful" tokens (like dynamic objects or uniform sky) can actually improve retrieval accuracy.
Key Results
WeiAD sets a new state-of-the-art benchmark across all evaluated datasets. When paired with WeiToP, it provides an unprecedented accuracy-efficiency trade-off curve.
- Outperforming CNN Baselines: At a token retention ratio of $\rho = 0.5$, WeiAD+WeiToP runs faster than MixVPR while delivering a 3.1% higher Recall@1. At $\rho = 0.4$, it becomes 0.29 ms faster per image than the fastest CNN baseline (Conv-AP) while boosting average Recall@1 by 5.8%.
- Superiority Over Generic Pruning: WeiToP consistently outperforms general vision token-pruning methods (such as ToMe, FastV, and DynamicViT), which suffer from severe performance drops when forced into aggressive pruning regimes.
Limitations and Future Work
While highly effective, the authors note a few areas for improvement:
- Fixed Hyperparameters: The cluster structure and tier partitions are currently fixed hyperparameters determined during training rather than being learned adaptively.
- Early Layer Semantic Blindness: Because token importance is estimated at an early layer to maximize speedups, the model might miss high-level semantic context needed for extremely challenging viewpoint changes.
- Architecture Constraints: The current formulation is strictly tailored for transformer-based backbones and cannot be directly applied to pure CNN architectures.
Summary updated