When Pixel Difference Patterns Meet ViT: PiDiViT for Few-Shot Object Detection

Hongliang Zhou, Yongxiang Liu, Canyu Mo, Weijie Li, Bowen Peng, Li LiuICCV 2025

TL;DR

This paper introduces PiDiViT, a novel Vision Transformer (ViT) architecture for few-shot object detection. PiDiViT addresses limitations in existing ViT-based methods by incorporating explicit modules to enhance pixel-level feature differences and capture multiscale information, achieving state-of-the-art results on COCO for few-shot, one-shot, and open-vocabulary object detection.

What Problem Does This Paper Address?

Few-shot object detection (FSOD) aims to identify objects from classes with very limited training examples. While recent approaches leverage powerful pretrained Vision Transformers (ViTs) to overcome data scarcity, they often overlook crucial low-level feature details. Specifically, existing methods can result in blurred object boundaries and smooth transitions between object centers and their edges, hindering accurate object-background distinction. Furthermore, effectively balancing the extraction of fine-grained local details with broader global contour information across multiple scales remains a challenge. This gap limits the performance of ViT-based models in few-shot scenarios.

Methods

The proposed PiDiViT builds upon the DE-ViT framework but introduces two key novel modules:

  • Difference Convolution Fusion Module (DCFM): This module enhances feature differences from object centers to boundaries. It achieves this by fusing pixel-wise central difference features (using a Central PDC block) with original features through an attention mechanism. This process sharpens target edges and improves discrimination between objects and backgrounds.
  • Multiscale Feature Fusion Module (MFFM): This module addresses the challenge of capturing information at various scales. It adaptively fuses features extracted by five different scale convolutional kernels (1x1, 3x3, 5x5, 7x7, and 3x3 dilated convolutions) using a scale attention mechanism. This allows the model to generate attention weights that balance the extraction of local details and global semantic information, leading to more robust object detection across diverse scenarios.

The overall architecture processes initial region proposals and ViT features through a three-layer fusion process incorporating these novel modules.

Datasets

The paper evaluates PiDiViT on the COCO dataset. The evaluation follows standard category splits for few-shot, one-shot, and open-vocabulary object detection tasks. For one-shot object detection, specific splits (Split-1/2/3/4) are used, with three parts serving as base classes and the remaining as novel classes.

Key Insights and Ideas

  • Pixel-level differences are crucial for few-shot learning: The paper highlights that even powerful pretrained ViTs can be improved by explicitly focusing on low-level feature discrepancies, particularly at object boundaries.
  • Bridging local and global information: The DCFM and MFFM modules effectively demonstrate how to combine fine-grained local details with broader contextual understanding, which is essential for robust object detection, especially with limited data.
  • Attention mechanisms are key for adaptive fusion: The use of attention in both DCFM and MFFM allows the model to dynamically weigh the importance of different features and scales, leading to more adaptive and effective feature representations.
  • ViT's potential is unlocked by specialized modules: PiDiViT shows that by designing specific modules tailored to address ViT's inherent limitations (like smooth boundaries), its powerful pretrained representations can be better leveraged for challenging tasks like few-shot detection.

Key Results

PiDiViT achieves state-of-the-art (SOTA) performance on the COCO benchmark across multiple few-shot learning scenarios:

  • Few-Shot Object Detection (FSOD): Surpasses existing SOTA by 2.7 nAP50 (10-shot) and 4.0 nAP50 (30-shot) for novel classes.
  • One-Shot Object Detection (OSOD): Exceeds SOTA by 4.4 nAP50 on average.
  • Open-Vocabulary Object Detection (OVD): Outperforms SOTA by 3.7 nAP50.

Qualitative comparisons show PiDiViT reduces duplicate detections, improves classification accuracy, and enhances detection completeness compared to the baseline DE-ViT.

Limitations and Future Work

The authors acknowledge that the inherent bias in pretrained ViTs can influence the projection of class-specific features, especially in scenarios with imbalanced class distributions.

As future work, they plan to design a fairness-constrained algorithm to dynamically adjust class weights during training. This aims to mitigate the impact of pretrained ViT bias and further optimize the performance balance between novel and base classes in imbalanced scenarios.

Summary updated

BibTeX
@inproceedings{zhou2025when,
  author = {Hongliang Zhou and Yongxiang Liu and Canyu Mo and Weijie Li and Bowen Peng and Li Liu},
  title = {{When Pixel Difference Patterns Meet ViT: PiDiViT for Few-Shot Object Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure