Latency Robust Cooperative Perception Using Asynchronous Feature Fusion

Junjie Wang, Tomas Nordström2025 · DOI

TL;DR

This paper introduces LRCP, a novel framework for cooperative perception that tackles the critical issue of latency. By directly fusing asynchronous Bird's-Eye View (BEV) features and employing a new flow prediction module, LRCP significantly improves robustness to temporal delays, achieving near-ideal performance even with latencies up to 500ms.

What Problem Does This Paper Address?

Cooperative perception, where multiple agents (like vehicles) share sensor data to build a more comprehensive understanding of their surroundings, offers significant advantages over single-agent systems. It can improve detection accuracy, handle occlusions, and reduce sensor noise. However, a major hurdle in real-world cooperative perception is latency. This delay, introduced by data processing and communication, causes features from different agents to be misaligned in time. Traditional methods often try to temporally align these features, but this process can be complex, prone to accumulating errors, and still leaves a performance gap compared to systems with no latency. This paper aims to bridge that gap by developing a method that can effectively fuse these inherently asynchronous features without requiring explicit temporal alignment.

Methods

The core of LRCP lies in its approach to handling asynchronous BEV features. Instead of trying to perfectly align features from different timestamps, LRCP directly fuses them. This is achieved through two key innovations:

  1. Reference Point Calibration with Deformable Attention: The framework utilizes deformable attention mechanisms, which are adept at sampling features from specific locations. For non-ego agents, the reference points used by this attention are "calibrated." This calibration involves predicting a 2D flow map that estimates how features from a past timestamp should be shifted to align with the current timestamp. This process is guided by a novel flow prediction module.
  2. Novel Flow Prediction Module: This module is designed to predict the flow between features from different timestamps. It takes a history of cached BEV features and the known latency as input. Crucially, it predicts flow at a non-discrete time delay, meaning it can handle arbitrary latency values, not just fixed intervals. This module leverages deformable attention to aggregate temporal information efficiently, avoiding the computational cost of 3D convolutions.

By directly using the predicted flow to adjust sampling locations within the attention mechanism, LRCP effectively synchronizes features at the point of fusion, minimizing error accumulation.

Datasets

The proposed LRCP framework was evaluated on two datasets:

  • V2X-Sim [19]: A simulated dataset built using SUMO and CARLA, designed for cooperative perception tasks. It includes multiple vehicles exchanging information.
  • Dair-V2X [35]: A real-world dataset featuring cooperative perception between a vehicle agent and an infrastructure agent.

For the flow prediction module, supervision was generated by projecting ground truth bounding boxes and calculating relative translations and yaw angles between objects in consecutive frames.

Key Insights and Ideas

  • Direct Fusion is Key: The paper argues that directly fusing asynchronous BEV features, rather than trying to create perfectly aligned features, is a more robust approach to handling latency. This avoids the error propagation inherent in complex alignment steps.
  • Flow Prediction for Calibration: Using a learned flow prediction module to calibrate reference points in attention mechanisms is a clever way to implicitly align features without explicit warping or complex temporal modeling.
  • Non-Discrete Flow Prediction: The ability to predict flow for any arbitrary time delay is crucial for real-world scenarios where latencies are not uniform or predictable.
  • Deformable Attention's Role: Deformable attention is well-suited for this task because it allows for dynamic sampling of features based on learned offsets, which can be guided by the predicted flow.

Key Results

LRCP demonstrated strong performance across various latency levels:

  • Robustness to Latency: The framework achieved robust performance with latencies up to 500 ms.
  • Minimal Degradation: At 500 ms latency, LRCP showed a performance degradation of only 1 percentage point for [email protected] and 4 percentage points for [email protected] on both V2X-Sim and Dair-V2X datasets. This is a significant improvement over existing methods.
  • Comparison to Baselines: LRCP outperformed other methods, including Deformable Attention (a strong baseline without calibration), SyncNet, CoBevflow, and V2X-ViT, especially under higher latencies.
  • Qualitative Improvements: Visualizations showed that LRCP effectively reduced inaccurate and duplicate detections that plagued other methods at higher latencies.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • V2X Communication Constraints: Real-world V2X communication faces bandwidth limitations, which could impact the amount of data shared.
  • Pose Estimation Noise: Pose estimation for moving agents can introduce noise, which might affect the accuracy of cooperative perception.
  • Ground Truth Flow Disparity: While LRCP performs well, there's still a slight performance gap compared to using ground truth flow, indicating room for improvement in the flow prediction module.
  • Privacy and Adversarial Robustness: Emerging concerns regarding data privacy and adversarial robustness in data sharing need further attention.
  • Flow Prediction Refinement: The authors observed minor inconsistencies (jitters) in flow predictions and plan to focus on developing strategies to refine these predictions.

Summary updated

Abstract

Recent advancements in cooperative perception have showcased substantial improvements compared to single-agent perception. Nonetheless, the inherent latency present in such systems can dramatically impair their effectiveness. In this paper, we propose a Latency Robust Cooperative Perception framework, named LRCP, to compensate for the effect of temporal asynchrony. The intuition of LRCP is to directly fuse asynchronous bird's-eye view (BEV) features instead of estimating aligned features. To achieve this, we first propose a novel flow prediction module that uses cached past BEV features to predict the flow with a non-discrete time delay at the BEV feature level. Then, the predicted flow is employed to guide the spatial sam-pling location of interests. Our approach substantially en-hances the robustness of temporal asynchronous cooper-ative perception. Specifically, we achieved robust performance across a range of latencies up to 500 ms, with a per-formance degradation of only 1 percent point for [email protected] metric and 4 percent points for [email protected] metric at 500ms on two public datasets (V2X-Sim and Dair- V2x). Code to reproduce our results is available at https://github.com/JesseWong333/LRCP.
BibTeX
@misc{wang2025latency,
  author = {Junjie Wang and Tomas Nordström},
  title = {{Latency Robust Cooperative Perception Using Asynchronous Feature Fusion}},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00476},
}
Figure