BoxMask: Revisiting Bounding Box Supervision for Video Object Detection

Khurram Azeem Hashmi, Alain Pagani, Didier Stricker, Muhammad Zeshan Afzal

Abstract

We present a new, simple yet effective approach to uplift video object detection. We observe that prior works operate on instance-level feature aggregation that imminently neglects the refined pixel-level representation, resulting in confusion among objects sharing similar appearance or motion characteristics. To address this limitation, we propose BoxMask, which effectively learns discriminative representations by incorporating class-aware pixel-level information. We simply consider bounding box-level annotations as a coarse mask for each object to supervise our method. The proposed module can be effortlessly integrated into any region-based detector to boost detection. Extensive experiments on ImageNet VID and EPIC KITCHENS datasets demonstrate consistent and significant improvement when we plug our BoxMask module into numerous recent state-of-the-art methods.

Introduction

Video Object Detection (VOD) faces unique challenges compared to still-image detection, including motion blur, rare object poses, and frequent occlusions. While contemporary VOD methods attempt to mitigate these issues by aggregating features across multiple frames (temporal aggregation), they typically operate at the instance level.

This instance-level focus often overlooks refined pixel-level details, leading to "object confusion." In such cases, models struggle to distinguish between objects with similar visual or motion characteristics—for instance, misidentifying a car as a watercraft—or fail to cleanly separate a moving object from a complex background. BoxMask addresses this by teaching models to learn pixel-level representations using only standard bounding box annotations.

Methodology

The core innovation is the BoxMask module, a "plug-and-play" component designed for integration into any region-based detector (e.g., Faster R-CNN or SELSA).

The BoxMask architecture integrates a pixel-level prediction head into standard region-based detectors, utilizing temporal RoI features from support frames.

The module operates through several key steps:

  1. Coarse Mask Supervision: Instead of requiring expensive pixel-perfect segmentation labels, BoxMask utilizes existing bounding box annotations. The area within each box is treated as a "coarse mask" for the object.
  2. Temporal RoI Feature Extraction: The module leverages features from "support frames" (nearby video frames) using cosine similarity and multi-head self-attention. This ensures that the pixel-level representation remains consistent across the temporal domain.
  3. BoxMask Head: A small, fully convolutional head is added to the network to predict a class-aware segmentation mask for each Region of Interest (RoI). This forces the network to learn discriminative, fine-grained features during training.
  4. Inference Efficiency: Crucially, the BoxMask head is only active during training. During inference, it is disabled, allowing for accuracy gains with virtually zero impact on detection speed.

Experimental Results

The authors evaluated BoxMask on two major benchmarks: ImageNet VID (30 classes) and EPIC KITCHENS (290 classes, egocentric video).

Qualitative comparison showing how BoxMask reduces object confusion and improves localization compared to standard VOD methods in challenging scenarios.

Key Findings:

  • State-of-the-Art Performance: When integrated with the TROI detector, BoxMask achieved 80.7% mAP on ImageNet VID using a ResNet-50 backbone.
  • Consistent Improvements: Across various base detectors and backbones (ResNet-50, ResNet-101, ResNeXt-101), mAP improvements ranged from +0.4% to +1.8%.
  • Enhanced Robustness: Qualitative analysis demonstrates that BoxMask effectively eliminates "hard false positives," such as background elements being misclassified as animals or vehicles.
  • Real-time Viability: The system maintains high frame rates (e.g., 13.0 FPS for SELSA+BoxMask on a DGX A100), making it suitable for practical applications.

Conclusion and Future Work

BoxMask demonstrates that bounding boxes already contain significant spatial knowledge that can be leveraged for pixel-level learning without additional manual labeling. By treating boxes as coarse masks, the model sharpens class boundaries and improves localization.

A current limitation is Faulty Mask Supervision, where background pixels within the rectangular bounding box (especially for diagonal objects) are treated as part of the mask. Future research may incorporate weakly supervised segmentation techniques, such as GrabCut, to automatically refine these coarse masks and further improve the training signal.

Summary updated

Abstract

We present a new, simple yet effective approach to uplift video object detection. We observe that prior works operate on instance-level feature aggregation that imminently neglects the refined pixel-level representation, resulting in confusion among objects sharing similar appearance or motion characteristics. To address this limitation, we propose BoxMask, which effectively learns discriminative representations by incorporating class-aware pixel-level information. We simply consider bounding box-level annotations as a coarse mask for each object to supervise our method. The proposed module can be effortlessly integrated into any region-based detector to boost detection. Extensive experiments on ImageNet VID and EPIC KITCHENS datasets demonstrate consistent and significant improvement when we plug our BoxMask module into numerous recent state-of-the-art methods.
BibTeX
@inproceedings{hashmi2023boxmask,
  author = {Khurram Azeem Hashmi and Alain Pagani and Didier Stricker and Muhammad Zeshan Afzal},
  title = {{BoxMask: Revisiting Bounding Box Supervision for Video Object Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=1B0qGhXLxV},
}
Figure