Abstract
We present a new, simple yet effective approach to uplift video object detection. We observe that prior works operate on instance-level feature aggregation that imminently neglects the refined pixel-level representation, resulting in confusion among objects sharing similar appearance or motion characteristics. To address this limitation, we propose BoxMask, which effectively learns discriminative representations by incorporating class-aware pixel-level information. We simply consider bounding box-level annotations as a coarse mask for each object to supervise our method. The proposed module can be effortlessly integrated into any region-based detector to boost detection. Extensive experiments on ImageNet VID and EPIC KITCHENS datasets demonstrate consistent and significant improvement when we plug our BoxMask module into numerous recent state-of-the-art methods.
Introduction
Video Object Detection (VOD) faces unique challenges compared to still-image detection, including motion blur, rare object poses, and frequent occlusions. While contemporary VOD methods attempt to mitigate these issues by aggregating features across multiple frames (temporal aggregation), they typically operate at the instance level.
This instance-level focus often overlooks refined pixel-level details, leading to "object confusion." In such cases, models struggle to distinguish between objects with similar visual or motion characteristics—for instance, misidentifying a car as a watercraft—or fail to cleanly separate a moving object from a complex background. BoxMask addresses this by teaching models to learn pixel-level representations using only standard bounding box annotations.
Methodology
The core innovation is the BoxMask module, a "plug-and-play" component designed for integration into any region-based detector (e.g., Faster R-CNN or SELSA).
The module operates through several key steps:
- Coarse Mask Supervision: Instead of requiring expensive pixel-perfect segmentation labels, BoxMask utilizes existing bounding box annotations. The area within each box is treated as a "coarse mask" for the object.
- Temporal RoI Feature Extraction: The module leverages features from "support frames" (nearby video frames) using cosine similarity and multi-head self-attention. This ensures that the pixel-level representation remains consistent across the temporal domain.
- BoxMask Head: A small, fully convolutional head is added to the network to predict a class-aware segmentation mask for each Region of Interest (RoI). This forces the network to learn discriminative, fine-grained features during training.
- Inference Efficiency: Crucially, the BoxMask head is only active during training. During inference, it is disabled, allowing for accuracy gains with virtually zero impact on detection speed.
Experimental Results
The authors evaluated BoxMask on two major benchmarks: ImageNet VID (30 classes) and EPIC KITCHENS (290 classes, egocentric video).
Key Findings:
- State-of-the-Art Performance: When integrated with the TROI detector, BoxMask achieved 80.7% mAP on ImageNet VID using a ResNet-50 backbone.
- Consistent Improvements: Across various base detectors and backbones (ResNet-50, ResNet-101, ResNeXt-101), mAP improvements ranged from +0.4% to +1.8%.
- Enhanced Robustness: Qualitative analysis demonstrates that BoxMask effectively eliminates "hard false positives," such as background elements being misclassified as animals or vehicles.
- Real-time Viability: The system maintains high frame rates (e.g., 13.0 FPS for SELSA+BoxMask on a DGX A100), making it suitable for practical applications.
Conclusion and Future Work
BoxMask demonstrates that bounding boxes already contain significant spatial knowledge that can be leveraged for pixel-level learning without additional manual labeling. By treating boxes as coarse masks, the model sharpens class boundaries and improves localization.
A current limitation is Faulty Mask Supervision, where background pixels within the rectangular bounding box (especially for diagonal objects) are treated as part of the mask. Future research may incorporate weakly supervised segmentation techniques, such as GrabCut, to automatically refine these coarse masks and further improve the training signal.
Summary updated