CorrDiff: Adaptive Delay-aware Detector with Temporal Cue Inputs for Real-time Object Detection

Xiang Zhang, Chenchen Fu, Yufei Cui, Lan Yi, Yuyang Sun, Weiwei Wu, Xue LiuarXiv 2025 · DOI · arXiv

Abstract

Real-time object detection in autonomous systems faces a critical challenge: streaming perception lag. Traditional models often assume a fixed processing delay, leading to "displacement errors" when actual system latency fluctuates. This paper introduces CorrDiff, a novel framework that adaptively predicts object locations based on real-time hardware and network conditions. By integrating a delay-aware scheduler with a temporal cue-based detector, CorrDiff achieves state-of-the-art performance across varying workloads and hardware configurations.

The Challenge of Streaming Perception Lag

In high-stakes applications like autonomous driving, the time taken to process a sensor frame can lead to dangerous inaccuracies. By the time a detection model outputs a bounding box, the target object has often moved from its original position. This phenomenon, known as streaming perception lag, creates a mismatch between the detected location and the real-world "now."

Overview of the streaming perception pipeline showing communication and computational delays.

While existing state-of-the-art (SOTA) models attempt to mitigate this by forecasting object positions one frame ahead, they typically assume a fixed delay. In real-world deployments, delays are dynamic due to fluctuating GPU workloads, background system tasks, and varying network bandwidth. A model hard-coded for a 33ms delay will fail when the system experiences a 50ms lag.

The CorrDiff Framework

CorrDiff addresses temporal uncertainty through two primary components:

  1. CDdetector: This specialized model utilizes "Temporal Cues" as inputs to understand object dynamics. It incorporates two sub-modules:
  2. Corr_Past: A spatial correlation sampler that measures feature movement between past frames, capturing object velocity and direction.
  3. Diff_Now: A module that calculates the temporal difference between current and previous frames to highlight motion and resolve visual ambiguities.
  4. CDscheduler: This scheduling algorithm monitors runtime statistics in real-time. Using an Exponential Moving Average (EMA), it estimates current communication and computational delays. It then directs the CDdetector to predict specific future timestamps ($C^F$) based on the observed lag.

To ensure robustness, the authors employed Mixed Speed Training, sampling various frame intervals during the training phase. This enables the model to handle a wide spectrum of scenarios, from slow-moving objects with minimal lag to high-speed motion under significant system stress.

Empirical Insights: The Non-Deterministic Nature of Delay

A key contribution of this work is the demonstration that system latency is inherently unpredictable. The authors observed that even on identical hardware, models like DAMO-StreamNet experience wildly different latencies depending on the concurrent system workload.

Analysis of system delays across different GPU servers, workloads, and bandwidths.

By treating the prediction interval as a dynamic input variable—a "temporal cue"—CorrDiff effectively decouples detection accuracy from the raw speed of the underlying hardware. Furthermore, the framework utilizes a suite of buffers (Historical Feature, Correlation, and Output buffers) to maintain high efficiency, allowing it to perform effectively even on older hardware like the RTX 2080Ti.

Performance and Results

The framework was evaluated primarily on the Argoverse-HD dataset, a benchmark for streaming perception featuring high-frequency (30 FPS) annotations.

  • State-of-the-Art Accuracy: CorrDiff achieved 38.1% sAP (streaming Average Precision) on Argoverse-HD, surpassing previous leaders such as DAMO-StreamNet and LongShortNet.
  • Hardware Agnostic Robustness: Unlike many competitors that degrade on less powerful devices, CorrDiff maintained superior perceptual accuracy across a range of GPUs, from the Tesla V100 to the RTX 2080Ti.
  • Adaptability to Speed: In "Acceleration Adaptation" tests simulating high-speed motion (up to 16x), CorrDiff showed an absolute improvement of up to 2.7% mAP over existing models.

Conclusion and Future Directions

CorrDiff represents a significant step forward in making real-time object detection resilient to the realities of system latency. While highly effective, the authors note that future work could involve deeper integration of temporal cues into initial feature learning stages. Additionally, explicitly modeling the external causes of workload fluctuations could lead to even more proactive and precise scheduling in complex autonomous environments.

Summary updated

Abstract

Real-time object detection takes an essential part in the decision-making process of numerous real-world applications, including collision avoidance and path planning in autonomous driving systems. This paper presents a novel real-time streaming perception method named CorrDiff, designed to tackle the challenge of delays in real-time detection systems. The main contribution of CorrDiff lies in its adaptive delay-aware detector, which is able to utilize runtime-estimated temporal cues to predict objects' locations for multiple future frames, and selectively produce predictions that matches real-world time, effectively compensating for any communication and computational delays. The proposed model outperforms current state-of-the-art methods by leveraging motion estimation and feature enhancement, both for 1) single-frame detection for the current frame or the next frame, in terms of the metric mAP, and 2) the prediction for (multiple) future frame(s), in terms of the metric sAP (The sAP metric is to evaluate object detection algorithms in streaming scenarios, factoring in both latency and accuracy). It demonstrates robust performance across a range of devices, from powerful Tesla V100 to modest RTX 2080Ti, achieving the highest level of perceptual accuracy on all platforms. Unlike most state-of-the-art methods that struggle to complete computation within a single frame on less powerful devices, CorrDiff meets the stringent real-time processing requirements on all kinds of devices. The experimental results emphasize the system's adaptability and its potential to significantly improve the safety and reliability for many real-world systems, such as autonomous driving. Our code is completely open-sourced and is available at https://anonymous.4open.science/r/CorrDiff.
BibTeX
@misc{zhang2025corrdiff,
  author = {Xiang Zhang and Chenchen Fu and Yufei Cui and Lan Yi and Yuyang Sun and Weiwei Wu and Xue Liu},
  title = {{CorrDiff: Adaptive Delay-aware Detector with Temporal Cue Inputs for Real-time Object Detection}},
  year = {2025},
  eprint = {2501.05132},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.05132},
}
Figure