Multimodal Transformer Model Using Time-Series Data to Classify Winter Road Surface Conditions

Yuya Moroto, Keisuke Maeda, Ren Togo, Takahiro Ogawa, Miki Haseyama

TL;DR

This paper introduces a novel multimodal Transformer model (MMTransformer) that leverages time-series data from road surface images and auxiliary information to accurately detect and predict winter road conditions. By employing cross-attention for sophisticated feature integration and incorporating temporal dynamics, the model significantly improves classification accuracy, offering a more robust solution for road safety management.

What Problem Does This Paper Address?

Winter road conditions, such as snow and ice, are a major cause of traffic accidents, particularly in regions like Japan where they account for a significant percentage of land area. Effectively managing these conditions requires accurate detection and prediction of road surface states to implement timely snow removal and anti-icing measures. Existing methods often rely on single data sources (like images) or simple integration of multiple modalities, which can be insufficient due to the complex and dynamic nature of winter road surfaces. There's a clear need for a more advanced approach that can effectively fuse heterogeneous data sources and capture temporal changes to improve the reliability of road condition classification.

Methods

The core of this work is the Multimodal Transformer (MMTransformer). The key innovations lie in:

  • Multimodal Feature Integration with Cross-Attention: Instead of simple concatenation, the model uses a cross-attention mechanism to fuse features extracted from road surface images and auxiliary data. This allows for mutual complementation between modalities, enhancing the representational power of the integrated features. Cross-attention helps the model learn correlations between different features and timesteps.
  • Time-Series Processing: The model processes data across multiple timesteps (e.g., 20-minute intervals) to capture the temporal evolution of road surface conditions. This is crucial because road conditions are not static and change over time.
  • Feature Extraction:
    • Visual Features: A pre-trained Vision Transformer (ViT) is fine-tuned to extract rich visual features from road surface images.
  • Auxiliary Features: Auxiliary data (temperature, road temperature, snowfall, traffic volume, weather forecasts) are processed. For discrete variables, a "soft label encoding" (SLE) is used instead of standard one-hot encoding to better capture correlations between features.
  • End-to-End Training: The entire model is trained end-to-end, allowing for joint optimization of feature extraction and classification.

Datasets

The study utilizes a dataset provided by the East Nippon Expressway Company Limited, collected between 2017 and 2019. This dataset includes:

  • Road Surface Images: Captured by fixed-point cameras at 20-minute intervals.
  • Auxiliary Data: Including temperature, road temperature, snowfall, traffic volume, vehicle speed, and weather forecasts at various future time points.

The original seven road surface condition categories (dry, wet, black sherbet, white sherbet, snow, compacted snow, ice) were reorganized into three broader categories for practical classification: dry/wet, sherbet, and snow/compacted snow/ice. Data from 2017-2018 was used for training, and 2019 data for testing.

Key Insights and Ideas

  • Temporal Dynamics Matter: Incorporating time-series data significantly improves performance, highlighting that road conditions are best understood as a dynamic process rather than isolated snapshots.
  • Cross-Attention is Key for Fusion: The cross-attention mechanism is crucial for effectively integrating heterogeneous modalities (images and auxiliary data), allowing them to "talk to each other" and compensate for each other's weaknesses.
  • Visualizing Predictions Enhances Trust: The supplemental experiments using image style transfer to visualize predicted road conditions offer a promising avenue for decision support, making the model's outputs more interpretable for road managers.
  • Subtle Differences are Challenging: The model struggles slightly more with differentiating between visually similar categories like "sherbet" and "snow/compacted snow/ice," suggesting that while multimodal data helps, visual ambiguity remains a challenge.

Key Results

The MMTransformer demonstrated strong performance in both detecting (0-hour prediction) and predicting (1-hour and 3-hour predictions) winter road surface conditions.

  • Time-Series Effectiveness: Increasing the number of timesteps used as input consistently led to higher macro F1 scores across all prediction horizons. For instance, in immediate detection, the macro F1 score increased from 0.702 (1 timestep) to 0.740 (5 timesteps).
  • Cross-Attention Advantage: The MMTransformer consistently outperformed a baseline "Concatenation" method (which simply concatenates features) in terms of macro F1 score. For 3-hour predictions, the MMTransformer achieved a macro F1 of 0.666, compared to 0.644 for the concatenation method.
  • Accuracy Metrics: Overall accuracy was high, often exceeding 90%, with macro F1 scores providing a more nuanced view of performance across different classes.
  • Qualitative Visualization: Attention rollout visualizations showed that the model focuses on relevant areas in the images, providing insights into how it makes its decisions. For example, it correctly identified a "sherbet" condition by attending to snow accumulation on the roadside while downplaying it on the road surface itself.

Limitations and Future Work

  • Visual Similarity: The paper acknowledges that performance improvement was slight for visually similar categories like "sherbet" and "snow/compacted snow/ice." Future work could focus on better leveraging non-visual information (auxiliary data) to overcome this visual ambiguity.
  • Later Predictions: While the model shows promise for predicting conditions a few hours ahead, improving accuracy for longer prediction horizons (e.g., 3 hours) remains a challenge.
  • Image Style Transfer Evaluation: The supplemental experiments on image generation demonstrated the potential of style transfer for visualization, but lacked quantitative evaluation and comparative analysis. Future work could involve developing specific image style transfer models for road surfaces and rigorously evaluating them.

Summary updated

Abstract

This paper proposes a multimodal Transformer model that uses time-series data to detect and predict winter road surface conditions. For detecting or predicting road surface conditions, the previous approach focuses on the cooperative use of multiple modalities as inputs, e.g., images captured by fixed-point cameras (road surface images) and auxiliary data related to road surface conditions under simple modality integration. Although such an approach achieves performance improvement compared to the method using only images or auxiliary data, there is a demand for further consideration of the way to integrate heterogeneous modalities. The proposed method realizes a more effective modality integration using a cross-attention mechanism and time-series processing. Concretely, when integrating multiple modalities, feature compensation through mutual complementation between modalities is realized through a feature integration technique based on a cross-attention mechanism, and the representational ability of the integrated features is enhanced. In addition, by introducing time-series processing for the input data across several timesteps, it is possible to consider the temporal changes in the road surface conditions. Experiments are conducted for both detection and prediction tasks using data corresponding to the current winter condition and data corresponding to a few hours after the current winter condition, respectively. The experimental results verify the effectiveness of the proposed method for both tasks. In addition to the construction of the classification model for winter road surface conditions, we first attempt to visualize the classification results, especially the prediction results, through the image style transfer model as supplemental extended experiments on image generation at the end of the paper.
BibTeX
@inproceedings{moroto2024multimodal,
  author = {Yuya Moroto and Keisuke Maeda and Ren Togo and Takahiro Ogawa and Miki Haseyama},
  title = {{Multimodal Transformer Model Using Time-Series Data to Classify Winter Road Surface Conditions}},
  booktitle = {Sensors},
  year = {2024},
  url = {https://openreview.net/forum?id=EQUPMLwG3v},
}
Figure