Abstract
Real-time object detection in autonomous systems faces a critical challenge: streaming perception lag. Traditional models often assume a fixed processing delay, leading to "displacement errors" when actual system latency fluctuates. This paper introduces CorrDiff, a novel framework that adaptively predicts object locations based on real-time hardware and network conditions. By integrating a delay-aware scheduler with a temporal cue-based detector, CorrDiff achieves state-of-the-art performance across varying workloads and hardware configurations.
The Challenge of Streaming Perception Lag
In high-stakes applications like autonomous driving, the time taken to process a sensor frame can lead to dangerous inaccuracies. By the time a detection model outputs a bounding box, the target object has often moved from its original position. This phenomenon, known as streaming perception lag, creates a mismatch between the detected location and the real-world "now."
While existing state-of-the-art (SOTA) models attempt to mitigate this by forecasting object positions one frame ahead, they typically assume a fixed delay. In real-world deployments, delays are dynamic due to fluctuating GPU workloads, background system tasks, and varying network bandwidth. A model hard-coded for a 33ms delay will fail when the system experiences a 50ms lag.
The CorrDiff Framework
CorrDiff addresses temporal uncertainty through two primary components:
- CDdetector: This specialized model utilizes "Temporal Cues" as inputs to understand object dynamics. It incorporates two sub-modules:
- Corr_Past: A spatial correlation sampler that measures feature movement between past frames, capturing object velocity and direction.
- Diff_Now: A module that calculates the temporal difference between current and previous frames to highlight motion and resolve visual ambiguities.
- CDscheduler: This scheduling algorithm monitors runtime statistics in real-time. Using an Exponential Moving Average (EMA), it estimates current communication and computational delays. It then directs the CDdetector to predict specific future timestamps ($C^F$) based on the observed lag.
To ensure robustness, the authors employed Mixed Speed Training, sampling various frame intervals during the training phase. This enables the model to handle a wide spectrum of scenarios, from slow-moving objects with minimal lag to high-speed motion under significant system stress.
Empirical Insights: The Non-Deterministic Nature of Delay
A key contribution of this work is the demonstration that system latency is inherently unpredictable. The authors observed that even on identical hardware, models like DAMO-StreamNet experience wildly different latencies depending on the concurrent system workload.
By treating the prediction interval as a dynamic input variable—a "temporal cue"—CorrDiff effectively decouples detection accuracy from the raw speed of the underlying hardware. Furthermore, the framework utilizes a suite of buffers (Historical Feature, Correlation, and Output buffers) to maintain high efficiency, allowing it to perform effectively even on older hardware like the RTX 2080Ti.
Performance and Results
The framework was evaluated primarily on the Argoverse-HD dataset, a benchmark for streaming perception featuring high-frequency (30 FPS) annotations.
- State-of-the-Art Accuracy: CorrDiff achieved 38.1% sAP (streaming Average Precision) on Argoverse-HD, surpassing previous leaders such as DAMO-StreamNet and LongShortNet.
- Hardware Agnostic Robustness: Unlike many competitors that degrade on less powerful devices, CorrDiff maintained superior perceptual accuracy across a range of GPUs, from the Tesla V100 to the RTX 2080Ti.
- Adaptability to Speed: In "Acceleration Adaptation" tests simulating high-speed motion (up to 16x), CorrDiff showed an absolute improvement of up to 2.7% mAP over existing models.
Conclusion and Future Directions
CorrDiff represents a significant step forward in making real-time object detection resilient to the realities of system latency. While highly effective, the authors note that future work could involve deeper integration of temporal cues into initial feature learning stages. Additionally, explicitly modeling the external causes of workload fluctuations could lead to even more proactive and precise scheduling in complex autonomous environments.
Summary updated