TL;DR
Researchers have developed TriCosmos-2.5, the first driving world model to jointly forecast RGB, thermal, and LiDAR data. By treating these three sensors as a single unified video stream and using specialized densification for LiDAR, the model significantly improves forecasting accuracy in adverse weather, reducing LiDAR error by 2.1 meters compared to single-sensor approaches.
What Problem Does This Paper Address?
Driving world models are designed to predict what a vehicle's sensors will see next, which is vital for training autonomous planners and anticipating road hazards. However, most existing models focus almost exclusively on RGB camera data.
In the real world, cameras are far from perfect. Fog, heavy snow, and darkness can render RGB sensors nearly blind. Thermal cameras (which detect heat) and LiDAR (which measures distance directly) are much more robust in these "adverse weather" conditions. Despite this, few models incorporate thermal data, and even fewer attempt to forecast all three modalities simultaneously.
The authors tackle three main hurdles:
- Representation: How do you feed sparse LiDAR points into a model designed for dense video pixels?
- Joint Learning: Does seeing thermal and LiDAR data actually help the model predict RGB better (and vice versa)?
- Evaluation Bias: Standard metrics often make models look worse than a simple "repeat the last frame" (persistence) baseline because the model's output is compressed by a lossy "tokenizer."
Methods
The team built upon Cosmos-Predict2.5, a 2-billion parameter latent diffusion transformer originally trained on RGB video. To adapt it for multi-modal driving, they used several clever engineering tricks:
- LiDAR as "Depth Video": They projected 3D LiDAR sweeps into the 2D plane of the RGB camera. To solve the sparsity problem (LiDAR only covers ~3% of pixels), they applied morphological closing and dilation to create a dense depth map that the model's video tokenizer could actually "understand."
- The Tri-Sensor Grid: Instead of building three separate models, they created a tri-sensor model that views a 2x2 grid containing RGB, thermal, and LiDAR depth tiles as a single video frame. This forces the model to learn the spatial and temporal correlations between sensors.
- LoRA Post-training: They used Low-Rank Adaptation (LoRA) to fine-tune the model on multi-modal data without retraining the entire 2B parameter backbone, keeping the base weights frozen.
- Compression-Matched Baselines: To ensure a fair fight, they compared their model against a "compression-matched persistence" baseline—essentially passing the last observed frame through the same lossy tokenizer the model uses before repeating it.
Datasets
The study utilizes the DSERT-RoLL dataset, which provides synchronized RGB, thermal, and LiDAR data across six real-world weather conditions: Clear, Fog, Light Rain, Heavy Rain, Light Snow, and Heavy Snow. * Training: 132 sequences (35 used for the final LoRA training). * Validation: 58 sequences used to benchmark performance across all weather types.
Key Insights and Ideas
- Joint Training is a Force Multiplier for LiDAR: One of the most striking findings is that a LiDAR-only model actually struggles to beat a simple persistence baseline. However, when trained jointly with RGB and thermal data, the LiDAR forecasts become significantly more accurate. The model uses the visual context from the cameras to "hallucinate" better depth.
- The "Compression Gap" Matters: In fog, the thermal model initially appeared to lose to the persistence baseline. However, the authors realized this was an artifact of the tokenizer. Once the baseline was "compression-matched," the model's superiority became clear.
- Weather-Dependent Reliance: The model doesn't rely on all sensors equally in all conditions. For example, blanking out the RGB sensor degrades LiDAR forecasts much more severely in snow than in fog, suggesting the model learns which sensors are most reliable for specific environments.
Key Results
- LiDAR Accuracy: The tri-sensor model reduced the Mean Absolute Error (MAE) of LiDAR forecasts at a 2-second horizon by 2.13 meters compared to a single-sensor LiDAR model (4.68m vs 6.81m).
- Beating Persistence: The tri-sensor model was the only one to beat the persistence baseline for LiDAR in mean error across almost all weather conditions.
- Visual Fidelity: Post-trained RGB and thermal forecasts consistently beat compression-matched persistence. For RGB, the Peak Signal-to-Noise Ratio (PSNR) rose by 2.42 dB over the baseline.
Limitations and Future Work
- Compute Constraints: Due to the high cost of generating diffusion-based forecasts, the authors only evaluated a 2-second horizon and trained on a subset of the full DSERT-RoLL dataset.
- Near-Range LiDAR: While the model excels at medium and far ranges, it still struggles to beat persistence at very close ranges (<10m), likely due to the densification process blurring fine details.
- Future Directions: The authors suggest scaling to longer horizons, using larger training subsets, and exploring different model backbones to see if these multi-modal synergies scale further.
Summary updated