EDF: Ensemble, Distill, and Fuse for Easy Video Labeling

Giulio Zhou, Subramanya Dulloor, David G. Andersen, Michael KaminskyarXiv 2018 · arXiv

Introduction

Training high-quality object detectors typically demands extensive manually labeled datasets. However, for video streams from fixed sources—such as surveillance cameras or recurring transit routes—general-purpose models often underperform due to the "domain gap." Lighting, camera angles, and specific occlusions in a local environment differ significantly from the broad datasets like MS-COCO used for pre-training. The EDF (Ensemble, Distill, and Fuse) framework addresses this by bootstrapping high-accuracy, video-specific detectors using a combination of model ensembling, distillation, and minimal human intervention.

The EDF Framework

The EDF methodology operates in three distinct stages to bridge the gap between generic detection and site-specific accuracy.

The EDF framework architecture, illustrating the flow from pre-trained detector ensembling to supervised label propagation and final fusion.
  1. Ensembling (The Generic Teacher): The system processes video frames through three off-the-shelf detectors: Faster R-CNN, SSD, and YOLOv2. A greedy Intersection-over-Union (IoU) merging strategy combines these outputs. A consensus-based reweighting scheme boosts confidence for boxes where models agree and treats disagreements with skepticism, creating a "cleaner" set of pseudo-labels.
  2. Distillation (The Student): These ensemble predictions serve as "soft targets" to train a lightweight Student Network (MobileNet-SSD with FPN). Heavy data augmentation, such as random cropping, is applied during training. This allows the student to learn a robust representation of objects within the specific video context, often surpassing the frame-by-frame performance of its "teachers."
  3. Fusion (The Specialized Labeler): To achieve peak performance, a tiny fraction of frames (0.5% to 3%) is manually labeled. A Supervised Labeler is trained on this subset. Finally, detections from the Student Network and the Supervised Labeler are fused to produce the final output.

Experimental Evaluation

The authors evaluated EDF on two benchmark datasets representing different challenges:

  • DETRAC: Stationary surveillance videos with high traffic density.
  • KITTI: First-person moving automotive videos with dynamic backgrounds.

Key Results and Insights

The most significant finding is that the distilled Student Network consistently outperforms the ensemble that generated its training labels. While the ensemble operates on a per-frame basis, the student is trained on the entire video sequence. Data augmentation enables the student to generalize better across the specific domain's visual variations.

Performance comparison of individual detectors, the ensemble, and the distilled student network across KITTI and DETRAC datasets.

As shown in the results, the distilled Student Network achieved a mean Average Precision (mAP) 1.25x and 1.21x higher than the best single ensemble member on DETRAC and KITTI, respectively. Furthermore, the EDF approach demonstrated remarkable label efficiency. On KITTI, combining the distilled student with just 3% human-labeled frames (approx. 236 frames) resulted in an mAP of 0.625, significantly higher than the ensemble's 0.451.

Conclusion

EDF provides a practical, cost-effective blueprint for specializing object detectors for specific video streams. By leveraging the collective knowledge of pre-trained models and refining it through distillation and sparse human labeling, the framework achieves high precision without the prohibitive costs of massive manual annotation. Future work aims to incorporate temporal consistency to further improve detection stability in video sequences.

Summary updated

BibTeX
@misc{zhou2018edf,
  author = {Giulio Zhou and Subramanya Dulloor and David G. Andersen and Michael Kaminsky},
  title = {{EDF: Ensemble, Distill, and Fuse for Easy Video Labeling}},
  year = {2018},
  eprint = {1812.03626},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.03626},
}
Figure