BoxMask: Revisiting Bounding Box Supervision for Video Object Detection

Khurram Azeem Hashmi, Alain Pagani, Didier Stricker, Muhammad Zeshan Afzal

Abstract

We present a new, simple yet effective approach to uplift video object detection (VOD). We observe that prior works operate on instance-level feature aggregation that imminently neglects the refined pixel-level representation, resulting in confusion among objects sharing similar appearance or motion characteristics. To address this limitation, we propose BoxMask, which effectively learns discriminative representations by incorporating class-aware pixel-level information. We simply consider bounding box-level annotations as a coarse mask for each object to supervise our method. The proposed module can be effortlessly integrated into any region-based detector to boost detection. Extensive experiments on ImageNet VID and EPIC KITCHENS datasets demonstrate consistent and significant improvement when we plug our BoxMask module into numerous recent state-of-the-art methods.

Introduction: Beyond Instance-Level Aggregation

Detecting objects in video is significantly more challenging than in static images due to factors like motion blur, rare poses, and frequent occlusions. While modern state-of-the-art VOD methods attempt to mitigate these issues through temporal feature aggregation, they typically operate at the instance level. By treating an entire bounding box as a single feature vector, these methods often fail to capture fine-grained pixel details. This lack of spatial precision leads to "object confusion"—where the model misidentifies objects with similar motion patterns (e.g., a blurry watercraft vs. a car) or mistakes background patches for objects.

The BoxMask Module

To address these limitations, we introduce BoxMask, a lightweight, "plug-and-play" module designed to enhance the discriminative power of region-based detectors like Faster R-CNN.

BoxMask architecture integrated into a standard region-based video object detector.

As shown in the architecture above, BoxMask adds a third task to the standard detection pipeline. In addition to classification and bounding box regression, the model learns to predict which pixels within a region belong to the object. The module consists of a simple head: a 3x3 convolution, a 2x2 deconvolution for upsampling, and a final 1x1 convolution.

Bounding Boxes as Coarse Masks

A key innovation of BoxMask is its supervision strategy. High-quality pixel-level segmentation masks are expensive and rare in video datasets. BoxMask bypasses this requirement by using existing bounding box annotations as coarse masks. Every pixel inside the box is treated as the object, while pixels outside are background. Although this supervision is "noisy" (as boxes contain background corners), it forces the network to learn more discriminative spatial features and ignore background noise within the Region of Interest (RoI).

Experimental Evaluation

We evaluated BoxMask on two major benchmarks: ImageNet VID and the challenging egocentric EPIC KITCHENS dataset.

Qualitative Improvements

The impact of pixel-level learning is most evident in cases of high ambiguity where baseline methods struggle with localization and classification.

Qualitative comparison showing how BoxMask reduces object confusion and improves localization precision compared to baseline VOD methods.

As illustrated in the comparison above, baseline methods often suffer from object confusion (e.g., misidentifying a trash can as a bear) or imprecise localization. By incorporating BoxMask, the detector becomes more confident in its classifications and more precise in its bounding box placement, effectively reducing "hard false positives."

Quantitative Results

BoxMask delivered consistent improvements across various baseline architectures (such as SELSA and TROI) and backbones:

  • ImageNet VID: Adding BoxMask to the TROI detector achieved a mean Average Precision (mAP) of 80.7% using a ResNet-50 backbone, setting a new state-of-the-art for that configuration.
  • EPIC KITCHENS: The module showed dramatic gains in high-precision metrics, with up to a 5.2 point increase in $AP_{0.75}$, demonstrating its robustness in complex, real-world environments.
  • Efficiency: The computational overhead is minimal. For the TROI detector, the inference speed only decreased slightly from 7.3 FPS to 7.2 FPS.

Conclusion and Future Work

BoxMask demonstrates that revisiting bounding box supervision through a pixel-level lens can significantly boost VOD performance without requiring additional manual annotations. While the "coarse mask" approach is highly effective, future work could involve refining these masks using unsupervised techniques to further bridge the gap between box-level and pixel-level supervision.

The code is available at https://github.com/khurramHashmi/BoxMask.

Summary updated

Abstract

We present a new, simple yet effective approach to up- lift video object detection. We observe that prior works operate on instance-level feature aggregation that imminently neglects the refined pixel-level representation, resulting in confusion among objects sharing similar appearance or motion characteristics. To address this limitation, we propose BoxMask, which effectively learns discriminative representations by incorporating class-aware pixel-level information. We simply consider bounding box-level annotations as a coarse mask for each object to supervise our method. The proposed module can be effortlessly integrated into any region-based detector to boost detection. Extensive experiments on ImageNet VID and EPIC KITCHENS datasets demonstrate consistent and significant improvement when we plug our BoxMask module into numerous recent state-of-the-art methods. The code will be available at https://github.com/khurramHashmi/BoxMask.
BibTeX
@inproceedings{hashmi2023boxmask,
  author = {Khurram Azeem Hashmi and Alain Pagani and Didier Stricker and Muhammad Zeshan Afzal},
  title = {{BoxMask: Revisiting Bounding Box Supervision for Video Object Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=1B0qGhXLxV},
}
Figure