TL;DR
This paper introduces an "Interactive Self-Annotation" (ISA) framework that significantly reduces the time and human effort required for annotating video object bounding boxes. By combining automatic self-supervised learning with interactive human feedback, ISA iteratively refines object detection models, leading to high-quality annotations with less manual labor.
What Problem Does This Paper Address?
Training deep learning models, especially for complex tasks like autonomous driving, requires vast amounts of labeled data. For video object detection, this means meticulously drawing bounding boxes around objects in every frame, a process that is both time-consuming and costly. Existing annotation tools often rely on manual input or basic propagation, which can be inefficient and struggle with occlusions or dynamic scenes. This paper tackles the challenge of creating high-quality bounding box annotations for video datasets more efficiently, aiming to bridge the gap between the data demands of deep learning and the practical limitations of manual annotation.
Methods
The core of the proposed framework is a two-phase approach:
- Automatic Recurrent Annotation (ARA): This phase uses self-supervised learning to train an object detector on unlabeled videos. The detector is repeatedly exposed to the same videos, iteratively improving its ability to generate pseudo ground-truth bounding boxes. A "Labeling Assistant" module plays a crucial role here, leveraging temporal consistency and tracking to refine these automatically generated boxes, making them of higher quality than previous iterations. This process essentially bootstraps a capable detector without human intervention.
- Interactive Recurrent Annotation (IRA): This phase incorporates a human annotator into the loop. The detector, trained by ARA, is presented to the annotator, who can correct any mistakes. A novel "Hierarchical Correction" module is introduced, which focuses on correcting keyframes rather than every single frame. The frame-distance for correction is adaptively decreased over iterations, allowing the system to efficiently leverage the detector's performance on neighboring frames. Human corrections are fed back to train the detector, guiding it towards more accurate annotations.
Datasets
The paper evaluates its framework on two datasets:
- Cityscapes: Used for evaluating the Automatic Recurrent Annotation (ARA) process.
- DAD (Dashcam Accident) dataset: Used for evaluating the full Interactive Self-Annotation (ISA) framework, particularly for its efficiency in interactive annotation.
The authors also mention using the MS-COCO and BDD datasets for pre-training their object detection models.
Key Insights and Ideas
- Self-Supervised Learning for Annotation: The idea of using a detector to "watch" unlabeled videos repeatedly to improve itself is a clever way to generate initial, albeit noisy, annotations without human input. This significantly reduces the burden on the human annotator.
- Hierarchical Correction for Efficiency: Instead of frame-by-frame correction, focusing on keyframes with a decreasing interval is a smart strategy. It leverages the temporal coherence of video and the detector's ability to generalize to nearby frames, drastically speeding up the interactive annotation process.
- Labeling Assistant for Quality: The Labeling Assistant module is critical for making the automatically generated pseudo-ground-truth useful. By combining detection results with tracking and temporal consistency, it filters out noise and improves the quality of initial labels.
- A New Metric: Similarity In Union (SIU): The authors propose SIU as a more appropriate metric for evaluating annotation tasks than traditional metrics like Average Precision (AP) or Intersection Over Union (IOU). SIU considers the global context and is less sensitive to noise, making it better suited for assessing the quality of bounding box generation in a broader sense.
Key Results
- Significant Time Savings: The ISA framework is reported to be up to 9x faster than manual annotation, annotating a video frame in an average of 19 seconds compared to 174 seconds for manual annotation.
- Reduced Human Labor: ISA cuts down human labor costs by reducing the number of corrected frames and objects. For example, it reduced corrected frames by 66.7% and corrected objects by 35.9% compared to manual annotation.
- Improved Annotation Quality: The framework achieves high-quality annotations, with ARA converging to mSIU and mIOU scores around 74% and 73.2% respectively on the Cityscapes dataset after three iterations.
- Outperforming Baselines: ISA outperforms other methods, including those using only pre-trained models or basic self-learning techniques, especially in terms of annotation efficiency and quality.
Limitations and Future Work
- Limitations: The paper acknowledges that the ARA process can still generate noisy detections, which is why the interactive component is crucial. The effectiveness of the Labeling Assistant depends on the quality of the initial detector and tracking methods.
- Future Work: The authors express interest in extending their framework to annotate not only bounding boxes but also masks of video objects in the future. They also aim to further explore domain adaptation capabilities.
Summary updated