Keeping Track of Vehicles While They Are Hidden

When a car's sensors lose sight of another vehicle, because a bus passes between them or a building blocks a cross street, today's driving stacks usually either drop that vehicle or slide its last box forward at constant speed. The same shortcut is used when a roadside sensor's report arrives late over a wireless link. Both are guesses about where a vehicle went while nobody could see it. We propose to make that guess well. The hidden vehicle's position should be carried as a small set of plausible futures, one per intent such as turning or going straight. Each future should be rolled forward through a motion model whose limits belong to that particular vehicle: how quickly it can brake, how sluggishly its engine responds. Those limits are estimated from the few seconds in which it was still visible. A loaded truck that disappears behind a bus cannot be where a hatchback could be two seconds later, and a forecast that knows this can be both tighter and more trustworthy.

The practical test is whether this belief holds up on real sensor data, not only in simulation. Real vehicle–infrastructure data gives an unusual opportunity here. When the car's own view is blocked, an elevated roadside sensor often still sees the hidden vehicle, so we can score the car's belief about an agent it could not observe against real ground truth. We therefore treat heterogeneous vehicle dynamics as the core mechanism and multi-modal intent as the output representation. Occlusion and delayed cooperative messages are the operating regime. The gap between simulated and real benchmarks is the evaluation lens, because simulators tend to give vehicles uniform, idealized dynamics, and a model that has learned those dynamics may be confidently wrong in real traffic.

Core Claim

For a vehicle that goes unobserved for 0.5 to 3 seconds, a forecast built as a mixture of intents is better than two alternatives: constant-velocity extrapolation and a class-generic kinematic forecaster. Each intent in the mixture is propagated through a longitudinal model with a per-vehicle actuator lag $\tau$ and per-vehicle acceleration and braking bounds, all inferred from the pre-occlusion track. The comparison covers three things. It should cover real data more reliably, meaning ground truth falls inside the predicted region at the stated rate. It should do so with smaller predicted regions. It should re-identify the vehicle more accurately when it reappears. We also claim the advantage over learned, class-generic models will be larger on real data (DAIR-V2X) than on simulated data (V2X-Sim, OPV2V). If this holds, part of the sim-to-real gap in cooperative perception comes from simulated dynamics being too uniform. It is not only a matter of rendering.

The longitudinal model is the first-order lag that the platooning literature uses to explain string instability in mixed fleets:

$$\dot a = \frac{u - a}{\tau}, \qquad u \in [u_{\min}, u_{\max}],$$

Here $a$ is the vehicle's actual acceleration and $u$ is the commanded acceleration. $\tau$ is the engine or brake response lag, and $[u_{\min}, u_{\max}]$ are the vehicle's braking and acceleration limits. This model sits under a kinematic bicycle model for lateral motion.

Relation to Prior Work

Two lines of work supply the method. Trajectron++ decodes control inputs from a latent variable that selects discrete intents, and integrates them through a unicycle model so forecasts are kinematically feasible. It reports up to 60% lower average displacement error than earlier generative methods. We reuse its decode-controls-then-integrate structure, but it uses one kinematic model per semantic class and assumes the agent's history is complete. We add per-vehicle lag and saturation parameters and we forecast through gaps in observation. The heterogeneous platoon controller in Distributed Model Predictive Control for Heterogeneous Vehicle Platoon with Unknown Input of Leading Vehicle gives each vehicle its own lag and input limits and shows this matters. It reports a tracking index of 4.3299 against 294.7686 for a benchmark law. However, it assumes those parameters are known and that neighbors' states are perfectly observed. We instead estimate the parameters from perception tracks of vehicles we do not control, which the research file lists as an open problem for that literature.

Other work motivates the proposal. Scene Informer infers hidden agents from how visible agents behave and reduces final displacement error by 2.37 m under limited observability on the Waymo Open Motion Dataset. Its ray-cast occlusion is simulated, it is egocentric only, and its paths carry no dynamic limits. We target a different case: an agent that was seen, then lost, then must be carried forward. DAIR-V2X shows that cooperative detection degrades when messages are delayed beyond 10 ms. Its Time Compensation Late Fusion, which shifts infrastructure boxes forward at estimated velocity, recovers up to 1.5% AP. We read that small gain as evidence that linear extrapolation leaves value unclaimed, though it may simply mean delays in that benchmark are short. We test both readings.

V2VNet and Where2comm share neural features between vehicles to remove blind spots. V2VNet reports roughly 60% better detection of heavily occluded objects and a 0.2 m forecast improvement at 3 s, all on simulated data. Where2comm states that it decides where to communicate in space but not when in time. Both constrain our design, because when communication works, much occlusion simply disappears. Our proposal therefore targets what remains. That residual includes regions no collaborator sees, dropped or stale messages, and the interval before a collaborator's view arrives. Finally, V2X-Sim finds that naive late fusion can hurt because it imports other vehicles' false positives. This is a warning that a stale report fused as a confident point can do harm. Our mixture representation is intended to down-weight such reports rather than trust them.

Research Program

The system has three parts. The first is a parameter inference head. From a vehicle's visible history, usually 1 to 3 seconds of tracked positions, it outputs a posterior over $(\tau, u_{\min}, u_{\max})$ and the vehicle's size class. It is trained on simulated tracks with dynamics randomized over a wide range and regularized toward physically plausible values, so it can express genuine uncertainty. For example, a vehicle observed only cruising says little about its braking limit, and the posterior should stay wide.

The second part is an intent-mixture forecaster. A Trajectron++-style decoder emits commanded controls $u$ for each of $K$ intent modes. We integrate them through the lag model and bicycle model under parameters sampled from the posterior, and clip them to the sampled bounds. The output is a mixture over intents in which each mode spreads according to parameter uncertainty. It is not a single path per mode.

The third part consumes the belief. During occlusion, the ego tracker holds the mixture as the hidden vehicle's state and uses it to match the vehicle when it reappears. When a delayed cooperative report arrives, the same forecaster propagates the report's objects over the measured delay before fusion. It replaces the constant-velocity shift of Time Compensation Late Fusion.

The analysis is designed so the heterogeneity claim can fail cleanly. We stratify every result by vehicle class and by whether the vehicle braked or turned during occlusion. That is where lag and braking limits should matter and where constant velocity should break. We expect little or no gain for cruising passenger cars in short occlusions, and we will report that.

Baselines

The naive reference is constant-velocity extrapolation with a Kalman filter and class-level process noise. It is essentially the motion model inside DAIR-V2X's Time Compensation Late Fusion, and it is what most cooperative pipelines use today. We tune its noise on the same training split with the same budget as our model. It is the reference every other method must beat, and for short occlusions of cruising cars it may be unbeatable.

The strongest existing forecaster is Trajectron++ run through occlusion gaps. It uses class-generic unicycle kinematics, gets the same pre-occlusion history, and is trained on the same simulated-plus-real data with a matched number of hyperparameter trials. We choose it because it already combines multi-modal intents with dynamics integration, so any gain we show cannot be credited merely to multi-modality or to having a physics layer. On the delayed-message task, the strongest existing reported number is Time Compensation Late Fusion's gain of up to 1.5% AP on asynchronous DAIR-V2X data, starting from 41.90 AP for synchronous LiDAR late fusion (DAIR-V2X). For occlusion-period forecast error on DAIR-V2X, we found no reported result on Lacuna, so our baselines establish that number.

The key ablation removes only the per-vehicle parameters. The architecture, data, and training are identical, but $(\tau, u_{\min}, u_{\max})$ are fixed per class. A second ablation keeps the parameters and removes the intent mixture ($K=1$). A third replaces the explicit dynamics layer with an unconstrained transformer decoder of matched parameter count. That transformer sees the same history and is free to infer vehicle type implicitly.

Beating the baselines means two things on the real test split. At matched coverage, meaning the fraction of ground-truth positions falling inside the predicted 90% region, the method must produce a smaller region area than every baseline. It must also produce lower negative log-likelihood. For heavy vehicles, and for vehicles that braked or turned while hidden, these gains should be statistically clear across three seeds. On the delay task, it must beat Time Compensation Late Fusion's AP at injected delays of 200 ms and above.

Why not just…?

Why not just use constant velocity with a Kalman filter? For occlusions under about half a second, or vehicles holding speed, this will probably match us, and we expect it to. The small 1.5% AP recovery that Time Compensation Late Fusion achieves on real DAIR-V2X suggests that at the benchmark's natural delays there is little to gain. The claim therefore lives in longer gaps and maneuvering vehicles. Our stratified results either show a widening advantage with gap length and maneuver intensity, or they show this alternative is sufficient. In the second case we would narrow the proposal to the maneuvering regime or drop it.

Why not just let a big learned forecaster infer vehicle type from history? This is the most serious threat. A transformer conditioned on a few seconds of motion may learn implicitly that slow-accelerating, long vehicles also brake gently. The unconstrained-decoder ablation settles it directly. Our specific bet is that implicit inference learned in simulation transfers worse. CARLA-style simulators give vehicles narrow, uniform dynamics, whereas an explicit parameter posterior trained under wide randomization should fail more gracefully on real trucks. If the unconstrained decoder matches us on both V2X-Sim and DAIR-V2X, the explicit dynamics layer is not earning its place.

Why not just share features over V2X so nothing is hidden? When it works, it does. V2VNet and Where2comm both show large gains on occluded objects, and V2X-Sim shows an elevated roadside unit lifting [email protected] from 70.43 to 77.08. Our method is complementary. It covers regions no collaborator sees, dropped packets, and stale messages. We evaluate it both with and without cooperative input to measure how much residual occlusion remains in real DAIR-V2X scenes. If that residual is negligible, the proposal's practical value is small.

Why not just randomize dynamics in simulation and train any forecaster on that? Domain randomization is part of our training recipe, not a rival to it. The research file notes that broad randomization tends to produce overly conservative behavior. In forecasting, that means predicted regions that are safe but wide. Per-vehicle inference is meant to take the benefit of randomization while narrowing the region when evidence allows. The comparison is class-fixed parameters trained under the same randomization against inferred parameters, and region area at matched coverage decides it.

Proposed Experiments

Training uses V2X-Sim, which provides annotated vehicle trajectories, and OPV2V, where cooperative views let us mark when an agent is hidden from one vehicle but visible to another. We randomize the simulated dynamics over lag and braking ranges. The real test set is DAIR-V2X cooperative pairs. There we define occlusion events as intervals in which an object is in the vehicle's range but has no vehicle-side detection, while the infrastructure side still observes it. Infrastructure observations serve as ground truth. A known risk is that the file does not say whether DAIR-V2X provides persistent object identities. If it does not, we build infrastructure-side tracks by association across its sequential frames, and we report association error so it is not confused with forecast error.

The metrics are minimum and mode-weighted displacement error over the occlusion interval, negative log-likelihood, coverage against area of the 90% region, and re-identification accuracy at reappearance. For the delay task, we report cooperative detection AP on DAIR-V2X with delays injected from 100 to 500 ms and random message dropouts.

We rely on three decisive checks. The first is the per-vehicle-parameter ablation on real heavy vehicles and on maneuvers during occlusion. That is where the mechanism must show its value or be abandoned. The second is the sim-to-real gap itself: the drop in coverage and NLL when moving from V2X-Sim to DAIR-V2X, compared across our model, the class-generic Trajectron++ baseline, and the unconstrained decoder. The claim requires our drop to be the smallest. The third is the delay curve. AP against injected delay should show our propagation separating from constant-velocity compensation as delay grows. If the curves overlap even at 500 ms, then in real intersection traffic delayed messages are not where intent and dynamics matter.