When several vehicles share what they see, each vehicle can follow nearby objects over time better than it could alone. The usual design is a one-way chain. Each vehicle detects objects, the vehicles exchange compressed sensor features, and only then does a tracker link detections across frames. That linking step, called data association, decides which new box belongs to which existing object track. The tracker is the first place where the system learns that it is confused: two cars are close together, a pedestrian has just come out from behind a truck, or a track has gone unobserved long enough that several detections could plausibly continue it. Under the current design, that confusion never travels back to change what is communicated. The tracker resolves it alone with a hard one-to-one assignment, and any mistake becomes an identity switch that downstream forecasting and planning treat as fact.
We propose to close this loop. The ego vehicle keeps a soft assignment between its predicted tracks and candidate detections, which gives a probability for each possible match instead of a single committed one. It measures how spread out that distribution is for each track. It then asks peer vehicles for information only in the regions where association is ambiguous. Because the tracker predicts where objects will be, these requests can go out before the next frame is sensed. The bet is that, for a fixed communication budget, "where am I unsure which object is which" is a better guide for what to request than "where is my detector unsure," the signal used by current bandwidth-efficient methods.
Core Claim
Requests driven by association ambiguity should cut identity switches and raise tracking accuracy at matched bandwidth, compared with requests driven by detection confidence or by track covariance alone. The gains should concentrate in crowded and occluded scenes, where confusion between nearby objects, not positional uncertainty about a single object, is what breaks the pipeline.
Each part of this claim is anchored in a specific prior result. Where2comm supplies the communication mechanism we reuse. It sends spatial request maps, the inverse of a detector's per-location confidence, to pull missing features from peers. Those maps are single-frame and see nothing of the tracker. PragComm adds foreground and keyframe selection and codebook compression, but it is also a detection-stage method. MOT-CUP is the closest work and supplies our association cost. It calibrates per-box variances with conformal prediction, a distribution-free method for producing intervals with coverage guarantees. It feeds those variances into a Kalman filter and a likelihood-based matching score on collaborative detections. Its pipeline is strictly feed-forward, however, and it collapses each frame to a hard Hungarian assignment, so ambiguity is never measured and never acted on.
UG3DMOT motivates the distinction between ambiguity and covariance. It shows that track covariance should widen the matching gate for long-occluded objects, and it reduced KITTI identity switches to 30, against 84 to 462 for competing trackers. It is single-agent, though, and covariance alone cannot tell an isolated lost car from two cars that could be swapped. DA-MCTS shows in single-agent planning that keeping association hypotheses alive and pruning them adaptively beats fixed heuristics: 90% success against 60%. We take from it the lesson that ambiguity should be kept and exploited rather than collapsed early. We apply that lesson to multi-agent information gathering instead of tree search.
Research Program
The tracker keeps Gaussian track states and receives detections that carry conformal variances, as in MOT-CUP. Instead of running the Hungarian algorithm directly, it first computes an entropic partial transport plan $P$ between tracks $i$ and detections $j$. Partial transport is a soft matching that is allowed to leave items unmatched. Following the formulation of Ratnayaka et al., we add an extra "unmatched" column for each track. The cost $C_{ij}$ is the negative log-likelihood of detection $j$ under track $i$'s predicted Gaussian. Normalizing each row of $P$ gives match probabilities $p_{ij}$, and the ambiguity of track $i$ is
$$a_i = -\sum_{j \in \{1..m,\varnothing\}} p_{ij}\log p_{ij},$$
the entropy of its match distribution, where $\varnothing$ denotes "no match." This score is high when several detections compete for a track, or when the track is torn between continuing and being lost. It stays low for an isolated object with a wide but uncontested gate. That separation is what distinguishes it from track covariance.
To form the request map for frame $t+1$, each track's predicted bird's-eye-view footprint is weighted by $a_i$, together with the footprints of the detections competing for it. Peers receive this map in place of, or blended with, the detector-confidence request of Where2comm. They respond with features restricted to the requested cells under a fixed byte budget, and fusion proceeds exactly as in Where2comm. The final assignment is still made with the Hungarian algorithm, so the output format is unchanged and differences can be attributed to what was communicated.
As a secondary contribution, we will also expose the soft marginals $p_{ij}$ to a simple trajectory forecaster. This tests whether keeping ambiguity, rather than collapsing it, also reduces downstream error. This part is exploratory and is not part of the core claim.
The analysis component splits every identity switch in the evaluation set into two kinds. A confusion event is one with a competing ground-truth object within the gate. A loss event is one with no competitor. We report all methods on both kinds, because the core claim predicts gains on confusion events specifically.
Baselines
The naive reference point is the ego vehicle tracking alone with no communication. Next to it is MOT-CUP-style tracking on top of Where2comm detections at the same byte budgets, with requests derived from detection confidence. This is the strongest existing system for the question, because it already combines calibrated uncertainty, likelihood-based association, and confidence-driven communication. Its only missing piece is the feedback we add.
MOT-CUP reports a 2% overall improvement in Higher Order Tracking Accuracy (HOTA, a metric that balances detection and identity correctness) on V2X-Sim, rising to 4.01% in high-occlusion scenes, along with a 1.03% MOTP gain. The file reports these only as relative improvements, with no absolute HOTA value. For Where2comm on V2X-Sim no reported result found on Lacuna, beyond the statement that it dominates the performance–bandwidth curve.
We will therefore rerun both on identical splits, with the same backbone detector, the same conformal calibration set, and the same tuning budget for gating thresholds. Results will be reported as HOTA and identity-switch curves against bytes per frame. Beating this baseline means a statistically reliable HOTA gain and identity-switch reduction at two or more budget levels across seeds, with most of the gain on confusion events.
The key ablation replaces $a_i$ with the trace of the track's predicted covariance and leaves everything else unchanged. This is the UG3DMOT signal turned into a request. It isolates whether ambiguity, rather than uncertainty in general, is the ingredient that matters. A second ablation keeps the soft plan but drops the anticipatory timing, computing requests from frame $t$ detections only, which separates the effect of prediction from the effect of the ambiguity score.
The upper references are raw-point early fusion, which DAIR-V2X prices at about 1.3 MB per frame, and an oracle that requests exactly the regions where ground-truth association errors occur.
Why not just…?
Why not just request where track covariance is large? This is the most serious threat, because covariance and ambiguity are correlated: occluded objects are both uncertain and easy to confuse. The covariance ablation settles it directly. If covariance-driven requests match ours on confusion events, the ambiguity score adds nothing and the contribution shrinks to "use the tracker, not the detector, to drive requests." That would still be a useful finding, and we would report it as such.
Why not just keep Where2comm's detector-confidence requests and smooth them over time? Detector confidence is high for two clearly visible cars that are close together, which is exactly the case where association fails. Temporal smoothing does not supply the missing "which is which" signal. We include a temporally smoothed Where2comm request as an additional baseline, and the confusion-versus-loss split will show whether it misses confusion events as predicted.
Why not just send more, or everything? Early fusion removes the ambiguity, but DAIR-V2X reports about 1.3 MB per frame against about 0.3 KB for late fusion. OPV2V finds that 4096× compression of intermediate features costs only about 3% AP. Bandwidth is therefore better spent selectively, and the question is where to spend it. Early fusion serves as our upper bound, not a competitor.
Why not just run multi-hypothesis tracking at the ego and wait for later frames to disambiguate? DA-MCTS shows that keeping hypotheses helps. Waiting, however, cannot recover information that no frame from the ego's viewpoint contains, and pruning in such methods is irreversible. We test this directly by giving the no-communication ego a multi-hypothesis tracker with a deferred decision window. If it closes most of the gap, communication-side feedback is less valuable than claimed.
Proposed Experiments
The primary benchmark is V2X-Sim, described in Li et al.. It has multi-agent LiDAR, tracking annotations, and roadside units, and it is the setting where MOT-CUP was evaluated. Sequential scenes in OPV2V and DAIR-V2X will serve as secondary testbeds where object identities can be recovered. We will report HOTA, identity switches, and MOTP against bytes per frame, the confusion-versus-loss decomposition, the 95% interval width and empirical coverage of tracks, and runtime overhead relative to MOT-CUP's reported 13.2%.
The decisive check is a single comparison at matched bandwidth among three request signals: ambiguity, covariance, and detector confidence. The core claim survives only if ambiguity wins on confusion-event identity switches and does not lose on loss events. A second check measures how much of the gap to the oracle request each signal closes. A third stratifies scenes by traffic density, where the advantage should grow with crowding.
References
- Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps
- Pragmatic Communication in Multi-Agent Collaborative Perception
- Collaborative Multi-Object Tracking With Conformal Uncertainty Propagation
- 3D Multi-Object Tracking Based on Uncertainty-Guided Data Association
- Data Association Aware POMDP Planning With Hypothesis Pruning Performance Guarantees
- Learning Partial Graph Matching via Optimal Partial Transport
- DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detection
- OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication
- V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Driving