Accelerated Video Annotation driven by Deep Detector and Tracker

Eric Price, Aamir AhmadarXiv 2023 · DOI · OpenReview

TL;DR

This paper introduces "Smarter-labelme," a novel video annotation tool that significantly speeds up the process of labeling objects in videos. By combining a deep learning-based detector (SSD-Multibox) with a recurrent deep tracker (Re³), it drastically reduces annotation drift and manual correction, leading to an average of 13% faster annotation speeds compared to baseline methods.

What Problem Does This Paper Address?

Accurately annotating object ground truth in videos is crucial for training and evaluating machine learning models in robotics and computer vision. However, manual annotation is a laborious and error-prone process. Existing automated methods often rely on classical tracking algorithms that can quickly "drift," meaning the tracked bounding box deviates from the actual object, necessitating frequent manual correction. This paper aims to address the time-consuming nature and high error rates associated with manual video annotation by developing a more robust and efficient automated approach.

Methods

The core of the proposed solution, "Smarter-labelme," is the integration of two key deep learning components:

  • SSD-Multibox Detector: This network is used for initial object detection and, crucially, to correct for drift introduced by the tracker. It operates on multi-scale tiled image regions to handle high-resolution videos without downscaling. A modified Non-Maximum Suppression (NMS) is employed to filter detections, prioritizing those with high confidence and avoiding re-detecting already annotated objects.
  • Re³ Tracker: This recurrent Siamese network uses Long-term-Short-term memory (LSTM) nodes to maintain object appearance information across frames. It predicts bounding box locations for subsequent frames.

The workflow involves using Re³ to predict an object's location in the next frame. Then, SSD-Multibox is used to re-detect the object. If the SSD detection has a high Intersection over Union (IOU) with the Re³ prediction (e.g., 0.8 or higher), the SSD detection is used to correct the Re³ prediction, and the label is copied. This drift compensation mechanism significantly improves tracking accuracy and reduces the need for manual intervention.

Datasets

The paper mentions using a set of drone video frames for annotation experiments. The SSD-Multibox detector was pre-trained on the MSCOCO dataset. No new datasets were introduced.

Key Insights and Ideas

  • Drift Compensation is Key: The paper highlights that the primary bottleneck in automated video annotation is tracker drift. By using a deep detector to actively correct the tracker's predictions, they achieve much higher accuracy and reduce manual effort.
  • Synergy of Detector and Tracker: The combination of a powerful detector (SSD) and a robust tracker (Re³) is more effective than either alone. The tracker provides a good initial guess, and the detector refines it, especially when drift occurs.
  • Handling High-Resolution Video: The multi-scale tiling approach for SSD allows the system to process high-resolution videos efficiently without sacrificing detection quality, which is a common challenge.
  • Manual Correction vs. Auto-Correction: The experiments revealed that while manual correction of tracker labels is faster than manual annotation from scratch, correcting errors from a poorly performing auto-annotation system can be slower than using a simple "copy-paste" baseline if the auto-annotation is unreliable.

Key Results

The experiments, conducted with 5 human annotators on a video of walking horses, showed that Smarter-labelme achieved an average of 13% faster annotation speeds compared to a baseline method (manual annotation with a pseudo-tracker). The OpenCV-based Smart-labelme was, on average, 3% slower than the baseline. The paper also notes that the quality of annotations is crucial; if pixel-exact annotations are required, the benefits of automated methods are more pronounced.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while Re³ suffers from drift, it only manifests after some time, still requiring occasional manual adjustment.
  • Annotators reported that correcting auto-annotations from SSD-Multibox in the first frame could be time-consuming if the initial detections were poor.
  • In complex scenes, certain objects might still cause trackers to fail, leading to consistently bad labels.
  • Future Work:
    • The authors suggest enabling annotators to disable the accelerated tracker for individual objects that are consistently mis-tracked.
  • Future work could involve online training of the detector in real-time, using the already labeled annotations as training data to further improve accuracy and adapt to specific object appearances.

Summary updated

Abstract

Annotating object ground truth in videos is vital for several downstream tasks in robot perception and machine learning, such as for evaluating the performance of an object tracker or training an image-based object detector. The accuracy of the annotated instances of the moving objects on every image frame in a video is crucially important. Achieving that through manual annotations is not only very time consuming and labor intensive, but is also prone to high error rate. State-of-the-art annotation methods depend on manually initializing the object bounding boxes only in the first frame and then use classical tracking methods, e.g., adaboost, or kernelized correlation filters, to keep track of those bounding boxes. These can quickly drift, thereby requiring tedious manual supervision. In this paper, we propose a new annotation method which leverages a combination of a learning-based detector (SSD) and a learning-based tracker (RE$^3$). Through this, we significantly reduce annotation drifts, and, consequently, the required manual supervision. We validate our approach through annotation experiments using our proposed annotation method and existing baselines on a set of drone video frames. Source code and detailed information on how to run the annotation program can be found at https://github.com/robot-perception-group/smarter-labelme
BibTeX
@misc{price2023accelerated,
  author = {Eric Price and Aamir Ahmad},
  title = {{Accelerated Video Annotation driven by Deep Detector and Tracker}},
  year = {2023},
  eprint = {2302.09590},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.09590},
}
Figure