Digital-Twin Losses for Lane-Compliant Trajectory Prediction at Urban Intersections

Kuo-Yi Chao, Erik Leo Haß, Melina Gegg, Jiajie Zhang, Ralph Raßhofer, Alois Knoll

TL;DR

Researchers from the Technical University of Munich and BMW have developed a trajectory prediction pipeline that uses "digital twin" data—specifically HD maps and infrastructure constraints—to guide neural networks. By incorporating these constraints as auxiliary training losses rather than model inputs, they achieved a 20% improvement in long-term prediction accuracy and significantly fewer traffic rule violations without increasing real-time inference latency.

What Problem Does This Paper Address?

Predicting where a vehicle or pedestrian will be in 5 seconds is relatively easy on a straight highway, but it’s a nightmare at busy urban intersections. Standard deep learning models often treat the world as an empty grid, occasionally predicting trajectories that "ghost" through medians or drive off the road because they don't inherently understand lane geometry.

While some state-of-the-art models solve this by feeding high-definition (HD) maps directly into the neural network (using complex structures like scene graphs), this adds massive computational overhead. Furthermore, the authors identify a subtle but critical "coordinate frame pitfall": trajectory models usually predict relative offsets (e.g., "move 2 meters left"), while maps use absolute coordinates. If you don't align these perfectly during training, the mathematical gradients become useless, and the model never learns to follow the road.

Methods

The authors propose a "pragmatic middle ground" using a standard Bi-LSTM (Bidirectional Long Short-Term Memory) encoder-decoder architecture. The magic happens in the training objective, which combines standard error-matching with a novel Twin Loss:

  1. Infrastructure Proximity Loss: This penalizes the model if a predicted path strays too far from the centerlines of the lanes defined in the digital twin's HD map.
  2. Collision Avoidance Loss: A "hinge penalty" that discourages the model from predicting trajectories for different agents that occupy the same space at the same time.
  3. Coordinate Recovery: To make these losses work, the authors implemented an "anchor recovery" step. They transform relative predictions back into absolute East-North-Up (ENU) coordinates before calculating the loss against the map. This ensures the gradients actually point the model toward the lanes.
  4. MC-Dropout: To handle uncertainty (e.g., will the car turn left or go straight?), they use Monte Carlo Dropout at inference time to generate 20 possible future paths.

Datasets

The study uses the TUMtraf V2X Cooperative Perception Dataset, which contains real-world data from a busy intersection in Munich.

  • Scale: ~90 minutes of 15Hz sensor data.
  • Complexity: Includes ~20,000 individual objects (cars, trucks, bicycles, pedestrians).
  • Preprocessing: The data was converted into ~1.14 million sliding-window samples for training and evaluation across prediction horizons ranging from 1 to 5 seconds.

Key Insights and Ideas

  • Loss as a Teacher, Not a Feature: By using map information only during training (as a loss function) and not as an input feature, the model learns "lane-keeping" behavior implicitly. This keeps the model lightweight and fast enough for real-time V2X applications.
  • The Coordinate Frame Fix: The authors demonstrate that without adding the "anchor" position back to relative predictions, map-based losses provide zero useful signal. This is a vital "gotcha" for anyone building map-aware AI.
  • Physics vs. Data: While simple physics models (like Constant Velocity) are great for the first 1 second, they fail miserably on curves. The "Twin Loss" allows the LSTM to outperform these classical models by 33% at the 2-second mark.

Key Results

  • Accuracy: The Twin_All model reached an Average Displacement Error (ADE) of 2.27m at a 5-second horizon, a 20% improvement over the baseline LSTM (2.84m) and nearly 42% better than classical Kalman filters (3.91m).
  • Safety: Infrastructure violations (driving off-lane) were reduced significantly. In their ablation study, correcting the coordinate frame alone improved lane-compliance by 17%.
  • Multimodality: Using MC-Dropout to generate multiple samples improved the "best-of-20" prediction accuracy by roughly 25% across all models.

Limitations and Future Work

Despite the improvements, the authors note several areas for growth:

  • Social Blindness: The current model predicts each agent's path independently. It doesn't "see" other agents as inputs, meaning it can't react to a car ahead slamming on its brakes (though the collision loss helps mitigate this during training).
  • Unimodal Latent Space: MC-Dropout is a "cheap" way to get multiple trajectories, but it doesn't explicitly model discrete intentions (like "Turn Left" vs. "Go Straight") as well as a Mixture Density Network (MDN) might.
  • Future Integration: The team plans to explore social interaction encoders and per-step velocity regression to further stabilize long-term predictions.

Summary updated

Abstract

Accurate and safety-conscious trajectory prediction is a key technology for intelligent transportation systems, especially in V2X-enabled urban environments with complex multi-agent interactions. In this paper, we created a digital twin-driven V2X trajectory prediction pipeline that jointly leverages cooperative perception from vehicles and infrastructure to forecast multi-agent motion at signalized intersections. The proposed model combines a Bi-LSTM-based generator with a structured training objective consisting of a standard mean squared error (MSE) loss and a novel twin loss. The twin loss encodes infrastructure constraints, collision avoidance, diversity across predicted modes, and rule-based priors derived from the digital twin. While the MSE term ensures point-wise accuracy, the twin loss penalizes traffic rule violations, predicted collisions, and mode collapse, guiding the model toward scene-consistent and safety-compliant predictions. We train and evaluate our approach on real-world V2X data sent from the intersection to the vehicle and collected in urban corridors. In addition to standard trajectory metrics (ADE, FDE), we introduce ITS-relevant safety indicators, including infrastructure and rule violation rates. Experimental results demonstrate that the proposed training scheme significantly reduces critical violations while maintaining comparable prediction accuracy and real-time performance, highlighting the potential of digital twin-driven multi-loss learning for V2X-enabled intelligent transportation systems.
BibTeX
@misc{chao2026digital,
  author = {Kuo-Yi Chao and Erik Leo Haß and Melina Gegg and Jiajie Zhang and Ralph Raßhofer and Alois Knoll},
  title = {{Digital-Twin Losses for Lane-Compliant Trajectory Prediction at Urban Intersections}},
  year = {2026},
  eprint = {2603.05546},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.05546},
}
Figure