Beyond the Camera: A Tri-Modal Driving World Model for Future RGB, Thermal, and LiDAR Forecasting in Adverse Weather

Faysal Mehrab Chowdhury, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Kishor Datta Gupta, Mohd Ariful Haque, Savannah Shannon

TL;DR

Researchers have developed TriCosmos-2.5, the first driving world model to jointly forecast RGB, thermal, and LiDAR data. By treating these three sensors as a single unified video stream and using specialized densification for LiDAR, the model significantly improves forecasting accuracy in adverse weather, reducing LiDAR error by 2.1 meters compared to single-sensor approaches.

What Problem Does This Paper Address?

Driving world models are designed to predict what a vehicle's sensors will see next, which is vital for training autonomous planners and anticipating road hazards. However, most existing models focus almost exclusively on RGB camera data.

In the real world, cameras are far from perfect. Fog, heavy snow, and darkness can render RGB sensors nearly blind. Thermal cameras (which detect heat) and LiDAR (which measures distance directly) are much more robust in these "adverse weather" conditions. Despite this, few models incorporate thermal data, and even fewer attempt to forecast all three modalities simultaneously.

The authors tackle three main hurdles:

  1. Representation: How do you feed sparse LiDAR points into a model designed for dense video pixels?
  2. Joint Learning: Does seeing thermal and LiDAR data actually help the model predict RGB better (and vice versa)?
  3. Evaluation Bias: Standard metrics often make models look worse than a simple "repeat the last frame" (persistence) baseline because the model's output is compressed by a lossy "tokenizer."

Methods

The team built upon Cosmos-Predict2.5, a 2-billion parameter latent diffusion transformer originally trained on RGB video. To adapt it for multi-modal driving, they used several clever engineering tricks:

  • LiDAR as "Depth Video": They projected 3D LiDAR sweeps into the 2D plane of the RGB camera. To solve the sparsity problem (LiDAR only covers ~3% of pixels), they applied morphological closing and dilation to create a dense depth map that the model's video tokenizer could actually "understand."
  • The Tri-Sensor Grid: Instead of building three separate models, they created a tri-sensor model that views a 2x2 grid containing RGB, thermal, and LiDAR depth tiles as a single video frame. This forces the model to learn the spatial and temporal correlations between sensors.
  • LoRA Post-training: They used Low-Rank Adaptation (LoRA) to fine-tune the model on multi-modal data without retraining the entire 2B parameter backbone, keeping the base weights frozen.
  • Compression-Matched Baselines: To ensure a fair fight, they compared their model against a "compression-matched persistence" baseline—essentially passing the last observed frame through the same lossy tokenizer the model uses before repeating it.
Figure 2. Overview. (a) RGB and thermal frames are cropped and resized to 1280 × 704 videos; LiDAR sweeps become a depth video aligned with the RGB camera. (b) The single-sensor models forecast each video on its own; the tri-sensor model forecasts a 2 × 2 grid of all three sensors, with the fourth tile left black. (c) Cosmos-Predict2.5 with low-rank adapters forecasts 20 frames from 5 observed frames. (d) Tri-sensor forecasts are split back into their sensors, and every forecast is scored against the real future.

Datasets

The study utilizes the DSERT-RoLL dataset, which provides synchronized RGB, thermal, and LiDAR data across six real-world weather conditions: Clear, Fog, Light Rain, Heavy Rain, Light Snow, and Heavy Snow. * Training: 132 sequences (35 used for the final LoRA training). * Validation: 58 sequences used to benchmark performance across all weather types.

Key Insights and Ideas

  • Joint Training is a Force Multiplier for LiDAR: One of the most striking findings is that a LiDAR-only model actually struggles to beat a simple persistence baseline. However, when trained jointly with RGB and thermal data, the LiDAR forecasts become significantly more accurate. The model uses the visual context from the cameras to "hallucinate" better depth.
  • The "Compression Gap" Matters: In fog, the thermal model initially appeared to lose to the persistence baseline. However, the authors realized this was an artifact of the tokenizer. Once the baseline was "compression-matched," the model's superiority became clear.
  • Weather-Dependent Reliance: The model doesn't rely on all sensors equally in all conditions. For example, blanking out the RGB sensor degrades LiDAR forecasts much more severely in snow than in fog, suggesting the model learns which sensors are most reliable for specific environments.

Key Results

  • LiDAR Accuracy: The tri-sensor model reduced the Mean Absolute Error (MAE) of LiDAR forecasts at a 2-second horizon by 2.13 meters compared to a single-sensor LiDAR model (4.68m vs 6.81m).
  • Beating Persistence: The tri-sensor model was the only one to beat the persistence baseline for LiDAR in mean error across almost all weather conditions.
  • Visual Fidelity: Post-trained RGB and thermal forecasts consistently beat compression-matched persistence. For RGB, the Peak Signal-to-Noise Ratio (PSNR) rose by 2.42 dB over the baseline.
Figure 1. Forecasting beyond the camera. From five observed frames, our models forecast the next 2 s of RGB, thermal, and LiDAR depth in real adverse weather. Shown: the last forecast frame (2 s) of one fog window. Columns: real future, two samples of the tri-sensor model, and the single-sensor model. The tri-sensor LiDAR forecasts stay close to the real depth, while the single-sensor LiDAR forecast is distorted. The camera forecasts of the two models look similar; in all of them, the vehicle on the left differs from the real future and between samples.

Limitations and Future Work

  • Compute Constraints: Due to the high cost of generating diffusion-based forecasts, the authors only evaluated a 2-second horizon and trained on a subset of the full DSERT-RoLL dataset.
  • Near-Range LiDAR: While the model excels at medium and far ranges, it still struggles to beat persistence at very close ranges (<10m), likely due to the densification process blurring fine details.
  • Future Directions: The authors suggest scaling to longer horizons, using larger training subsets, and exploring different model backbones to see if these multi-modal synergies scale further.

Summary updated

Abstract

Driving world models forecast what a vehicle's sensors will record next. Most forecast camera video and some add LiDAR, but thermal cameras, which still see in darkness and some adverse weather, are rarely a forecast target, and to our knowledge no world model forecasts all three sensors. We study joint forecasting of RGB, thermal, and LiDAR on real adverse-weather sequences of the DSERT-RoLL dataset. We post-train a pretrained video world model for each sensor and represent LiDAR as a depth video aligned with the RGB camera, with densification and encoding chosen by what the model's video tokenizer can reconstruct. A tri-sensor model sees all three sensors as one video. We score LiDAR in meters against measured returns and compare against persistence baselines passed through the same tokenizer as the model. With this matching, an apparent loss of the thermal model to persistence in fog disappears. On all 58 validation sequences in six weather conditions, post-trained RGB and thermal forecasts beat compression-matched persistence. Joint training lowers the LiDAR error at 2s by 2.1m at a small cost in camera fidelity, and only the tri-sensor model's LiDAR forecasts beat persistence in mean error. Blanking one sensor degrades the forecasts of the others more in snow than in fog.
BibTeX
@misc{chowdhury2026beyond,
  author = {Faysal Mehrab Chowdhury and Md. Mahfuzur Rahman and Ahmed Rafi Hasan and Kishor Datta Gupta and Mohd Ariful Haque and Savannah Shannon},
  title = {{Beyond the Camera: A Tri-Modal Driving World Model for Future RGB, Thermal, and LiDAR Forecasting in Adverse Weather}},
  year = {2026},
  eprint = {2611.22669},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22669},
}
Figure