Crash Injury Severity Prediction Using an Ordinal Classification Machine Learning Approach

Shengxue Zhu, Ke Wang, Chongyi LiInternational Journal of Environmental Research and Public Health 2021 · DOI

TL;DR

This paper introduces a novel ordinal classification machine learning approach to predict traffic crash injury severity, addressing the limitations of existing methods that ignore the inherent order of injury levels and struggle with imbalanced datasets. By combining category simplification, oversampling, and a unique probability calibration and thresholding strategy, their method significantly outperforms both nominal and existing ordinal classification techniques, offering more reliable predictions for critical injury levels.

What Problem Does This Paper Address?

Traffic crash injury severity prediction is crucial for safety analysis and intervention. However, existing machine learning approaches often treat injury severity levels as independent categories (nominal classification), ignoring the natural ordering (e.g., minor injury < moderate injury < severe injury). This simplification leads to suboptimal predictions. Furthermore, crash data is notoriously imbalanced, with a disproportionately high number of non-injury or minor injury cases compared to severe or fatal ones. This imbalance can cause models to be biased towards the majority class, failing to accurately predict the critical minority classes. Existing ordinal classification methods also suffer from issues like rank inconsistency and non-monotonicity, further hindering their effectiveness. This paper aims to bridge this gap by proposing an ordinal classification method that respects the ordered nature of injury severity, effectively handles class imbalance, and ensures rank consistency and monotonicity.

Methods

The core of this research lies in a novel ordinal classification framework. The approach involves several key steps:

  • Category Combination: To mitigate class imbalance and simplify the problem, the original five injury severity levels (non-injury crash, complaint of pain, other visible injury, severe injury, killed) are combined into three classes: minor injury, moderate injury, and serious injury. The paper explores six different combination strategies to find the most effective one.
  • SMOTE-NC Oversampling: To further address the class imbalance, the Synthetic Minority Oversampling Technique for Nominal and Continuous (SMOTE-NC) is employed. This technique generates synthetic samples for minority classes, balancing the dataset without losing valuable information from the majority classes.
  • Ordinal Binary Decomposition: The ordinal target variable is decomposed into multiple binary variables using a one-vs-all approach. This allows for the training of multiple single-output classifiers.
  • Probability Calibration: Predicted probabilities from the classifiers can be biased due to class imbalance and sampling methods. The paper utilizes isotonic regression to calibrate these probabilities, ensuring they more accurately reflect the true likelihood of each severity level.
  • Optimal Threshold Moving: Instead of using a default threshold (e.g., 0.5) to convert probabilities into class predictions, the authors propose finding an optimal threshold that maximizes the F1 score for each class. This is crucial for improving the prediction of minority classes.

The paper compares standard classifiers like MLP, XGBoost, and SVM for the initial prediction task, ultimately selecting XGBoost as the best performer for the subsequent ordinal classification steps.

Datasets

The study utilizes data collected from the Highway Safety Information System (HSIS) for crashes that occurred in California in 2010. The dataset comprises crash, vehicle, and occupant files, merged to create a comprehensive dataset. After cleaning, 139,555 crash records were used for analysis, with 17 key influencing variables identified.

Key Insights and Ideas

  • Ordinality Matters: The paper strongly advocates for treating injury severity as an ordinal variable, demonstrating that ignoring this inherent order leads to significantly poorer predictions, especially for critical injury levels.
  • Category Simplification is Key: Reducing the five original severity levels to three (minor, moderate, serious) substantially improves prediction precision, particularly for the most severe categories.
  • Calibrated Probabilities are Crucial: Raw probabilities from machine learning models, especially tree-based ones, can be biased. Calibrating these probabilities using methods like isotonic regression is essential for accurate ordinal classification and reliable thresholding.
  • Optimal Thresholds Outperform Defaults: Using default thresholds (like 0.5) for binary classification within the ordinal framework leads to suboptimal performance. Finding F1-maximizing thresholds for each class significantly boosts prediction accuracy, especially for minority classes.
  • Feature Importance Reveals Critical Factors: The analysis highlights that factors like "ejected from vehicle," "number of vehicles involved," "occupant type," and "vehicle model year" are consistently important predictors of crash injury severity across different category combinations.

Key Results

  • The proposed ordinal classification method achieved the highest precision rate for class 3 (serious injury) at 41.2%, with a recall of 21.3% and an F1 score of 28.1%. This is a significant improvement over existing methods.
  • Statistical significance tests (5x2 cross-validation with paired Student's t-test) confirmed that the proposed method is statistically superior to nominal classification and other ordinal classification benchmarks.
  • The proposed method demonstrated minimal increase in computational cost compared to nominal classification, with probability calibration and threshold moving adding only about 4.5 seconds to the XGBoost training time.
  • When comparing different category combination strategies, combination 2 (grouping COP and OVI into class 2, and SI and KSI into class 3) was found to be the most effective for a 3-class problem after SMOTE-NC sampling.
  • The optimal thresholds found (T₁=0.43, T₂=0.33) led to significantly better performance for class 3 compared to default thresholds (T₁=0.5, T₂=0.5), with precision increasing from 15.8% to 41.2%.

Limitations and Future Work

  • Limitations: The paper acknowledges that even with category combination, the class imbalance for the most severe categories (SI and KSI) remains a challenge, leading to lower precision rates for these classes compared to minor injury classes. The performance of Beckham's methods was also negatively impacted by class imbalance.
  • Future Work: The authors suggest several avenues for future research:
  • Integrating cost-sensitivity into the ordinal classification framework to further enhance its practical utility.
  • Developing methods to solve the 5-category classification problem directly without combining categories, which would offer finer-grained predictions.
  • Exploring other machine learning algorithms and advanced techniques for ordinal classification.

Summary updated

Abstract

In many related works, nominal classification algorithms ignore the order between injury severity levels and make sub-optimal predictions. Existing ordinal classification methods suffer rank inconsistency and rank non-monotonicity. The aim of this paper is to propose an ordinal classification approach to predict traffic crash injury severity and to test its performance over existing machine learning classification methods. First, we compare the performance of the neural network, XGBoost, and SVM classifiers in injury severity prediction. Second, we utilize a severity category-combination method with oversampling to relieve the class-imbalance problem prevalent in crash data. Third, we take advantage of probability calibration and the optimal probability threshold moving to improve the prediction ability of ordinal classification. The proposed approach can satisfy the rank consistency and rank monotonicity requirement and is proved to be superior to other ordinal classification methods and nominal classification machine learning by statistical significance test. Important factors relating to injury severity are selected based on their permutation feature importance scores. We find that converting severity levels into three classes, minor injury, moderate injury, and serious injury, can substantially improve the prediction precision.
BibTeX
@article{zhu2021crash,
  author = {Shengxue Zhu and Ke Wang and Chongyi Li},
  title = {{Crash Injury Severity Prediction Using an Ordinal Classification Machine Learning Approach}},
  journal = {International Journal of Environmental Research and Public Health},
  year = {2021},
  doi = {10.3390/ijerph182111564},
}
Figure