DATNet: Dense Auxiliary Tasks for Object Detection

Alex Levinshtein, Alborz Rezazadeh Sereshkeh, Konstantinos DerpanisWACV 2020

TL;DR

This paper introduces DATNet, a novel object detection architecture that combines the strengths of two-stage detectors (accuracy) with single-stage detectors (simplicity). By leveraging dense, per-pixel auxiliary tasks like semantic segmentation and distance transform, DATNet enriches the features used for object detection, leading to significant accuracy improvements over existing methods on challenging benchmarks.

What Problem Does This Paper Address?

Object detection is a cornerstone of computer vision, with applications ranging from autonomous driving to medical imaging. While two-stage detectors (like Faster R-CNN and Mask R-CNN) have historically achieved state-of-the-art accuracy, they are often computationally expensive and complex to optimize due to their region proposal and subsequent processing stages. In parallel, single-stage, anchor-free methods have emerged, offering simpler and faster architectures by framing detection as dense, per-pixel prediction tasks. However, these methods sometimes lag behind two-stage approaches in accuracy. The authors aim to bridge this gap by combining the best of both worlds: the robust feature representation of two-stage methods with the efficiency and simplicity of dense prediction tasks.

Methods

DATNet builds upon the Mask R-CNN architecture but introduces several key innovations:

  • Dense Auxiliary Tasks: The core idea is to train the network not only for object detection but also for auxiliary tasks that provide complementary information. Specifically, they use:
  • Distance Transform (DT): Predicts the distance from each pixel to the nearest object instance boundary. This helps in delineating object shapes.
  • Semantic Segmentation: Predicts the class of each pixel. This provides contextual class information. These auxiliary tasks are trained using dense, per-pixel predictions across the entire image.

  • Feature Fusion: Features from the backbone network are fused with the features generated by the auxiliary task branches. This fusion happens early in the pipeline, enriching the backbone features before they are used for region proposal and subsequent detection.

  • Multi-Scale Attention Pooling (MSPwA): Instead of assigning proposals to a single feature map level (as in standard FPN), DATNet concatenates features from multiple FPN levels. An attention mechanism (specifically, a Squeeze-and-Excitation module) is then applied to re-weight the channels, allowing the network to dynamically focus on the most informative features across different scales.
  • Modified ROI Head: The authors replace the standard ROI head with a ResNet Conv5 block, which is better suited to leverage the richer, fused features and the multi-scale information provided by MSPwA.

Datasets

The paper evaluates DATNet on two widely used datasets:

  • COCO 2017: A large-scale dataset with 80 object categories, used for extensive ablation studies and comparisons.
  • Pascal VOC 2007 & 2012: Used to demonstrate the effectiveness of DATNet even when instance segmentation ground truth is unavailable, by using bounding boxes as a proxy for coarse instance segmentation.

Key Insights and Ideas

  • Auxiliary tasks as feature enrichers, not just losses: The paper emphasizes that the value of auxiliary tasks lies not just in their individual losses but in how their learned features can be fused to create a more discriminative representation for the primary task.
  • Early fusion is key: Fusing auxiliary task features with backbone features early in the network allows these enriched features to influence the region proposal network (RPN) and subsequent stages, leading to better proposals and detections.
  • Attention for multi-scale feature utilization: The MSPwA module is crucial for effectively combining features from different FPN levels. The attention mechanism allows the network to intelligently select and weigh features, overcoming the limitations of hard scale assignments.
  • Leveraging instance segmentation data for semantic tasks: The authors show that they can generate semantic segmentation ground truth from instance segmentation annotations, making their approach adaptable to datasets that might not have explicit semantic labels but do have instance masks.

Key Results

  • Significant accuracy improvements: DATNet achieves substantial gains in both bounding box (AP) and mask (APM) detection metrics on the COCO dataset, outperforming strong baselines like Mask R-CNN and PANet. For example, DATNet Heavy with ResNeXt-101 and multi-scale training (MST) achieves a box AP of 44.7, a notable improvement over Mask R-CNN's 42.6.
  • Effectiveness without instance segmentation: On Pascal VOC, DATNet shows that using bounding boxes as a proxy for instance segmentation still yields performance gains, outperforming Mask R-CNN.
  • Ablation studies confirm contributions: The paper presents detailed ablation studies showing that both auxiliary tasks and the MSPwA module contribute positively to performance, with their combination yielding the best results.
  • Trade-offs in head design: The "Heavy" ROI head architecture, while achieving higher box AP, comes with increased computational cost compared to the "Light" version.

Limitations and Future Work

  • Inference Speed: DATNet, particularly the "Heavy" variant, has a longer inference time compared to Mask R-CNN due to the more complex ROI head and multi-scale pooling. The authors acknowledge this trade-off between accuracy and speed.
  • RPN Fusion Complexity: While the paper explores more complex feature fusion strategies for the RPN, they note that their simple fusion strategy already yields competitive results, suggesting that further research into advanced fusion for proposal generation could be beneficial.
  • Mask Prediction Improvement: The authors mention that while their focus is object detection, improvements to mask prediction are left for future work, particularly for the "Light" head architecture.

Summary updated

BibTeX
@inproceedings{levinshtein2020datnet,
  author = {Alex Levinshtein and Alborz Rezazadeh Sereshkeh and Konstantinos Derpanis},
  title = {{DATNet: Dense Auxiliary Tasks for Object Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure