Real-time Object Detection for Streaming Perception

Jinrong Yang, Songtao Liu, Zeming Li, Xiaoping Li, Jian Sun

Abstract

Autonomous driving requires the model to perceive the environment and (re)act within a low latency for safety. While past works ignore the inevitable changes in the environment after processing, streaming perception is proposed to jointly evaluate the latency and accuracy into a single metric for video online perception. In this paper, instead of searching trade-offs between accuracy and speed like previous works, we point out that endowing real-time models with the ability to predict the future is the key to dealing with this problem. We build a simple and effective framework for streaming perception. It equips a novel Dual-Flow Perception module (DFP), which includes dynamic and static flows to capture the moving trend and basic detection feature for streaming prediction. Further, we introduce a Trend-Aware Loss (TAL) combined with a trend factor to generate adaptive weights for objects with different moving speeds. Our simple method achieves competitive performance on Argoverse-HD dataset and improves the AP by 4.9% compared to the strong baseline, validating its effectiveness.

Introduction: The Latency Trap

In the world of autonomous driving, speed isn't just a performance metric—it's a safety requirement. However, even the fastest object detectors suffer from a hidden "latency trap": by the time a model finishes processing a video frame, the world has already moved on. Standard object detection benchmarks are typically "offline," assuming the world freezes while the computer thinks. In reality, if a self-driving car takes 30ms to detect a pedestrian, that pedestrian will have moved by the time the system reacts.

Baseline performance illustrating the 'latency trap' where detections (red) lag behind the actual object positions (green) due to processing time.

A recent metric called streaming accuracy (sAP) was proposed to address this. It evaluates the detection results of frame $t$ against the ground truth of the frame that exists when the computation finishes. Most high-performance detectors see a massive "performance cliff" when moved from offline to streaming settings because their predictions are always one step behind.

Methodology: Forecasting the Future

The authors propose that the key to streaming perception is endowing real-time detectors with the ability to forecast. They build StreamYOLO on the YOLOX architecture, introducing three main innovations:

  1. The Triplet Training Strategy: Instead of training on a single frame, the model is trained on triplets: it takes the previous frame ($F_{t-1}$) and the current frame ($F_t$) as input to predict the ground truth of the next frame ($G_{t+1}$).
  2. Dual-Flow Perception (DFP): To fuse information from the two input frames, the DFP module utilizes a Dynamic Flow to capture the moving trend (the "delta") and a Static Flow (a residual connection) to preserve basic semantic features, ensuring the model tracks stationary objects effectively.
  3. Trend-Aware Loss (TAL): Recognizing that fast-moving objects are harder to predict, TAL calculates a "trend factor" based on the Intersection over Union (IoU) between consecutive ground truth boxes. It adaptively assigns higher weights to the loss of fast-moving objects, forcing the model to prioritize difficult trends.

Results and Performance

The framework was evaluated on Argoverse-HD, a high-frame-rate (30 FPS) dataset designed for streaming perception in urban environments.

StreamYOLO results showing accurate alignment between predictions and object locations by forecasting the future frame.

StreamYOLO achieved a significant accuracy boost, improving the sAP of a YOLOX-L baseline from 31.2% to 36.1% (+4.9%). Key insights from the evaluation include:

  • Efficiency: By using a feature buffer to store the previous frame's features, the model adds a negligible 0.8ms of latency compared to the base detector.
  • Robustness: When testing with artificially accelerated video (2x speed), StreamYOLO remained robust, widening its lead over standard methods as the environment became more dynamic.
  • Superiority over Traditional Methods: The end-to-end learnable forecasting outperformed traditional Kalman Filter-based approaches by 0.6 sAP while maintaining higher speeds.

Conclusion

StreamYOLO demonstrates that treating real-time detection as a forecasting task is a powerful solution for the latency challenges in autonomous perception. While a gap still exists between offline and streaming accuracy—particularly for objects newly emerging in the future frame—this work provides a simple, effective, and real-time framework for safer online perception.

Summary updated

Abstract

Autonomous driving requires the model to perceive the environment and (re)act within a low latency for safety. While past works ignore the inevitable changes in the environment after processing, streaming perception is proposed to jointly evaluate the latency and accuracy into a single metric for video online perception. In this paper, instead of searching trade-offs between accuracy and speed like previous works, we point out that endowing real-time models with the ability to predict the future is the key to dealing with this problem. We build a simple and effective frame-work for streaming perception. It equips a novel Dual-Flow Perception module (DFP), which includes dynamic and static flows to capture the moving trend and basic detection feature for streaming prediction. Further, we introduce a Trend-Aware Loss (TAL) combined with a trend factor to generate adaptive weights for objects with different moving speeds. Our simple method achieves competitive performance on Argoverse-HD dataset and improves the AP by 4.9% compared to the strong baseline, validating its effectiveness. Our code will be made available at https://github.com/yancie-yjr/StreamYOLO.
BibTeX
@inproceedings{yang2022real,
  author = {Jinrong Yang and Songtao Liu and Zeming Li and Xiaoping Li and Jian Sun},
  title = {{Real-time Object Detection for Streaming Perception}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  url = {https://openreview.net/forum?id=KM2IyuoUef},
}
Figure