TL;DR
Object detectors often "flicker" in videos, missing an object for just a frame or two despite clear visibility. This paper identifies that a significant portion of these "Momentarily Missed Detections" (MMD) is caused by the rigid way anchor-based detectors are trained. By replacing hard IOU thresholds with a "soft thresholding" approach during training, the authors significantly improve temporal consistency in videos without sacrificing overall detection accuracy.
What Problem Does This Paper Address?
Even the best modern object detectors, like SSD or Faster R-CNN, suffer from a frustrating phenomenon in video: Momentarily Missed Detection (MMD). This occurs when an object is correctly detected in frames $t-1$ and $t+1$, but completely disappears in frame $t$.
While some misses are caused by external factors like motion blur or occlusion (see Figure 2), the authors discovered that 20–30% of cases have no obvious visual cause. They hypothesized that these are due to internal factors—specifically, how detectors handle the boundaries between "anchor boxes" (the predefined candidate boxes the model uses to find objects).
Methods
To investigate, the authors developed a novel analysis technique: Image Warping. They took frames where MMD occurred and applied tiny geometric transformations (scaling, shifting) to simulate smooth motion.
They found that as an object moves or changes size, the responsibility for detecting it passes from one anchor box to its neighbor. At the exact boundary between these anchors, the confidence scores of both anchors often "dip," creating a valley where the detection falls below the threshold.
To fix this, they proposed a Soft Sampling Method. Standard training uses a binary threshold (e.g., if IOU > 0.5, it's a positive sample; otherwise, it's negative). This creates "sharp" decision boundaries. The authors replaced this with a logistic sigmoid function, assigning continuous weights to anchor boxes based on their IOU. This encourages the network to produce more stable, overlapping confidence peaks, filling in the "valleys" at the boundaries.
Datasets
- DAVIS 2017: Used for the core analysis of MMD in video. It contains 90 videos with high-quality pixel-wise masks, which the authors converted into ground-truth bounding boxes.
- PASCAL VOC 2007 + 2012: Used for training the detectors (SSD and M2Det) and evaluating standard Mean Average Precision (mAP).
Key Insights and Ideas
- The "Valley" Effect: The most striking insight is that MMD is often a structural byproduct of anchor-based designs. When an object is exactly halfway between two anchors, neither anchor feels "responsible" enough to give it a high score.
- Internal vs. External: By categorizing misses, the authors showed that while we can't easily fix motion blur (external) within the detector's architecture, we can fix the internal boundary issues.
- Softness is Stability: Moving from "hard" binary labels to "soft" continuous weights during training makes the detector more robust to small changes in an object's position or scale, which is critical for video.
Key Results
- MMD Reduction: The soft sampling method dramatically reduced internal MMD cases. For an SSD-VGG16 model, MMD frames caused by anchor boundaries dropped from 73 to 11.
- Unexpected Bonus: Interestingly, the soft sampling also reduced some MMD cases attributed to external factors (like blur), likely because the increased robustness helps the model "hang on" to a detection even when the image quality degrades slightly.
- Maintained Accuracy: Crucially, this improvement in temporal stability did not hurt the standard mAP. In fact, mAP slightly improved in most tests (e.g., from 76.5 to 77.2 for SSD-VGG16).
Limitations and Future Work
- External Factors Remain: While the method helps, the majority of MMD (60-70%) is still caused by external factors like heavy blur or cluttered backgrounds. The authors suggest these might require specific pre-processing steps (like deblurring) or temporal models (like RNNs/LSTMs) to solve completely.
- Aspect Ratio Boundaries: The authors noted they didn't find many MMD cases caused by aspect ratio boundaries, likely because objects in the DAVIS dataset change size and position much more frequently than they change their fundamental shape.
- Hyperparameter Tuning: The "softness" of the sigmoid function ($\alpha$ and $\beta$) was chosen intuitively; further optimization might yield even better results.
Summary updated