Analysis and a Solution of Momentarily Missed Detection for Anchor-based Object Detectors

Yusuke Hosoya, Masanori Suganuma, Takayuki Okatani

TL;DR

Object detectors often "flicker" in videos, missing an object for just a frame or two despite clear visibility. This paper identifies that a significant portion of these "Momentarily Missed Detections" (MMD) is caused by the rigid way anchor-based detectors are trained. By replacing hard IOU thresholds with a "soft thresholding" approach during training, the authors significantly improve temporal consistency in videos without sacrificing overall detection accuracy.

What Problem Does This Paper Address?

Even the best modern object detectors, like SSD or Faster R-CNN, suffer from a frustrating phenomenon in video: Momentarily Missed Detection (MMD). This occurs when an object is correctly detected in frames $t-1$ and $t+1$, but completely disappears in frame $t$.

An example of MMD where a person and bicycle are missed in the middle frame despite no obvious visual change.

While some misses are caused by external factors like motion blur or occlusion (see Figure 2), the authors discovered that 20–30% of cases have no obvious visual cause. They hypothesized that these are due to internal factors—specifically, how detectors handle the boundaries between "anchor boxes" (the predefined candidate boxes the model uses to find objects).

Methods

To investigate, the authors developed a novel analysis technique: Image Warping. They took frames where MMD occurred and applied tiny geometric transformations (scaling, shifting) to simulate smooth motion.

The three types of anchor boundaries: scale, grid (position), and aspect ratio.

They found that as an object moves or changes size, the responsibility for detecting it passes from one anchor box to its neighbor. At the exact boundary between these anchors, the confidence scores of both anchors often "dip," creating a valley where the detection falls below the threshold.

To fix this, they proposed a Soft Sampling Method. Standard training uses a binary threshold (e.g., if IOU > 0.5, it's a positive sample; otherwise, it's negative). This creates "sharp" decision boundaries. The authors replaced this with a logistic sigmoid function, assigning continuous weights to anchor boxes based on their IOU. This encourages the network to produce more stable, overlapping confidence peaks, filling in the "valleys" at the boundaries.

Datasets

  • DAVIS 2017: Used for the core analysis of MMD in video. It contains 90 videos with high-quality pixel-wise masks, which the authors converted into ground-truth bounding boxes.
  • PASCAL VOC 2007 + 2012: Used for training the detectors (SSD and M2Det) and evaluating standard Mean Average Precision (mAP).

Key Insights and Ideas

  • The "Valley" Effect: The most striking insight is that MMD is often a structural byproduct of anchor-based designs. When an object is exactly halfway between two anchors, neither anchor feels "responsible" enough to give it a high score.
A case study showing how detection scores for two neighboring anchors (blue and magenta) both drop at the boundary (n=0), causing a miss.
  • Internal vs. External: By categorizing misses, the authors showed that while we can't easily fix motion blur (external) within the detector's architecture, we can fix the internal boundary issues.
  • Softness is Stability: Moving from "hard" binary labels to "soft" continuous weights during training makes the detector more robust to small changes in an object's position or scale, which is critical for video.

Key Results

  • MMD Reduction: The soft sampling method dramatically reduced internal MMD cases. For an SSD-VGG16 model, MMD frames caused by anchor boundaries dropped from 73 to 11.
  • Unexpected Bonus: Interestingly, the soft sampling also reduced some MMD cases attributed to external factors (like blur), likely because the increased robustness helps the model "hang on" to a detection even when the image quality degrades slightly.
  • Maintained Accuracy: Crucially, this improvement in temporal stability did not hurt the standard mAP. In fact, mAP slightly improved in most tests (e.g., from 76.5 to 77.2 for SSD-VGG16).

Limitations and Future Work

  • External Factors Remain: While the method helps, the majority of MMD (60-70%) is still caused by external factors like heavy blur or cluttered backgrounds. The authors suggest these might require specific pre-processing steps (like deblurring) or temporal models (like RNNs/LSTMs) to solve completely.
  • Aspect Ratio Boundaries: The authors noted they didn't find many MMD cases caused by aspect ratio boundaries, likely because objects in the DAVIS dataset change size and position much more frequently than they change their fundamental shape.
  • Hyperparameter Tuning: The "softness" of the sigmoid function ($\alpha$ and $\beta$) was chosen intuitively; further optimization might yield even better results.

Summary updated

Abstract

The employment of convolutional neural networks has led to significant performance improvement on the task of object detection. However, when applying existing detectors to continuous frames in a video, we often encounter momentary miss-detection of objects, that is, objects are undetected exceptionally at a few frames, although they are correctly detected at all other frames. In this paper, we analyze the mechanism of how such miss-detection occurs. For the most popular class of detectors that are based on anchor boxes, we show the followings: i) besides apparent causes such as motion blur, occlusions, background clutters, etc., the majority of remaining miss-detection can be explained by an improper behavior of the detectors at boundaries of the anchor boxes; and ii) this can be rectified by improving the way of choosing positive samples from candidate anchor boxes when training the detectors.
BibTeX
@inproceedings{hosoya2020analysis,
  author = {Yusuke Hosoya and Masanori Suganuma and Takayuki Okatani},
  title = {{Analysis and a Solution of Momentarily Missed Detection for Anchor-based Object Detectors}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=70BlIlvRgDO},
}

Loading related content...

Figure