Abstract
Autonomous driving requires the model to perceive the environment and (re)act within a low latency for safety. While past works ignore the inevitable changes in the environment after processing, streaming perception is proposed to jointly evaluate the latency and accuracy into a single metric for video online perception. In this paper, instead of searching trade-offs between accuracy and speed like previous works, we point out that endowing real-time models with the ability to predict the future is the key to dealing with this problem. We build a simple and effective framework for streaming perception. It equips a novel Dual-Flow Perception module (DFP), which includes dynamic and static flows to capture the moving trend and basic detection feature for streaming prediction. Further, we introduce a Trend-Aware Loss (TAL) combined with a trend factor to generate adaptive weights for objects with different moving speeds. Our simple method achieves competitive performance on Argoverse-HD dataset and improves the AP by 4.9% compared to the strong baseline, validating its effectiveness.
Introduction: The Latency Trap
In the world of autonomous driving, speed isn't just a performance metric—it's a safety requirement. However, even the fastest object detectors suffer from a hidden "latency trap": by the time a model finishes processing a video frame, the world has already moved on. Standard object detection benchmarks are typically "offline," assuming the world freezes while the computer thinks. In reality, if a self-driving car takes 30ms to detect a pedestrian, that pedestrian will have moved by the time the system reacts.
A recent metric called streaming accuracy (sAP) was proposed to address this. It evaluates the detection results of frame $t$ against the ground truth of the frame that exists when the computation finishes. Most high-performance detectors see a massive "performance cliff" when moved from offline to streaming settings because their predictions are always one step behind.
Methodology: Forecasting the Future
The authors propose that the key to streaming perception is endowing real-time detectors with the ability to forecast. They build StreamYOLO on the YOLOX architecture, introducing three main innovations:
- The Triplet Training Strategy: Instead of training on a single frame, the model is trained on triplets: it takes the previous frame ($F_{t-1}$) and the current frame ($F_t$) as input to predict the ground truth of the next frame ($G_{t+1}$).
- Dual-Flow Perception (DFP): To fuse information from the two input frames, the DFP module utilizes a Dynamic Flow to capture the moving trend (the "delta") and a Static Flow (a residual connection) to preserve basic semantic features, ensuring the model tracks stationary objects effectively.
- Trend-Aware Loss (TAL): Recognizing that fast-moving objects are harder to predict, TAL calculates a "trend factor" based on the Intersection over Union (IoU) between consecutive ground truth boxes. It adaptively assigns higher weights to the loss of fast-moving objects, forcing the model to prioritize difficult trends.
Results and Performance
The framework was evaluated on Argoverse-HD, a high-frame-rate (30 FPS) dataset designed for streaming perception in urban environments.
StreamYOLO achieved a significant accuracy boost, improving the sAP of a YOLOX-L baseline from 31.2% to 36.1% (+4.9%). Key insights from the evaluation include:
- Efficiency: By using a feature buffer to store the previous frame's features, the model adds a negligible 0.8ms of latency compared to the base detector.
- Robustness: When testing with artificially accelerated video (2x speed), StreamYOLO remained robust, widening its lead over standard methods as the environment became more dynamic.
- Superiority over Traditional Methods: The end-to-end learnable forecasting outperformed traditional Kalman Filter-based approaches by 0.6 sAP while maintaining higher speeds.
Conclusion
StreamYOLO demonstrates that treating real-time detection as a forecasting task is a powerful solution for the latency challenges in autonomous perception. While a gap still exists between offline and streaming accuracy—particularly for objects newly emerging in the future frame—this work provides a simple, effective, and real-time framework for safer online perception.
Summary updated