Application of Machine Learning Method in Simulation Model Validation

Qi Lin, Yong Chen2018 · DOI

Abstract

Simulation model validation often suffers from a lack of quantitative rigor, leading to a persistent distrust in simulation results. This paper proposes a machine learning-based framework to provide measurable and comparable judgments of simulation correctness. By adapting Leave-One-Out Cross-Validation (LOOCV) and incorporating an AdaBoost-inspired scaling factor, the method predicts the fit of a simulation model against a validation dataset. The approach is demonstrated on a Tethered Satellite System simulation, yielding a model correctness score of 91.4%, thereby providing a robust mathematical basis for validation.

Introduction: The Challenge of Simulation Trust

In modern engineering, simulations are indispensable tools for design and analysis. However, a fundamental gap exists between simulation outputs and real-world reliability. Traditional validation methods are often qualitative or lack standardized metrics, resulting in a "distrust" of simulation results.

The primary challenge lies in "data hunger": while simulation data is abundant, high-fidelity real-world data is scarce and expensive. Over-reliance on simulation data leads to overfitting, while insufficient real data makes error estimates unreliable. This research addresses these issues by framing simulation validation as a machine learning problem, specifically focusing on Generalization Error—the measure of how well a model performs on unseen, real-world data.

Methodology: Quantitative Validation via Cross-Validation

The core of the proposed framework is the application of Leave-One-Out Cross-Validation (LOOCV). In this process, the dataset is partitioned such that the model is iteratively trained on all but one data point and tested on the remaining point. This ensures that every piece of scarce experimental data is utilized effectively. The authors mathematically demonstrate that the average error from LOOCV provides an almost unbiased estimate of the out-of-sample error.

Proposed quantitative validation framework

The framework utilizes two primary data sources:

  • $D_{val}$: High-fidelity experimental data or "prototype" system data (the ground truth).
  • $D_{sim}$: Data generated by the simulation model being tested.

To optimize the validation process, an AdaBoost-inspired scaling factor is introduced. This factor re-weights the data points: weights for reliable experimental data are boosted to increase their influence, while weights for simulation data are scaled down. This "boosted" approach ensures that the validation process prioritizes ground truth while leveraging the volume of simulated data for better coverage and computational efficiency.

Experimental Setup and Results

The methodology was tested using a Tethered Satellite System simulation, specifically a first-order J2 perturbation propagator model. The dataset comprised 1,534 records, including 850 high-fidelity records ($D_{val}$) and 684 simulation records ($D_{sim}$). The features analyzed included velocity and position across three dimensions (Vx, Vy, Vz, x, y, z).

Calculated error sequence for simulation parameters

The validation process calculated quantitative error scopes for each parameter. For instance, the error for velocity in the X-direction (E-Vx) was found to range between 0.01 and 0.085. By aggregating these results, the maximum recorded error across all parameters was 0.086. This resulted in a calculated model correctness of 91.4%. In the context of satellite engineering, this score exceeds the standard industry requirement of 85%, providing a rigorous mathematical justification for trusting the simulation.

Conclusion and Future Directions

This research demonstrates that machine learning techniques, particularly cross-validation and adaptive scaling, can provide a robust, quantitative framework for simulation validation. By treating a simulation model as a hypothesis to be tested for generalization, engineers can move beyond qualitative "gut feelings" to concrete correctness percentages.

While the computational intensity of LOOCV remains a challenge for very large datasets, the use of scaling factors significantly improves efficiency. Future work will focus on automating the selection of heuristic parameters and extending the method to handle more chaotic or unstable simulation environments where random uncertainty is more prevalent.

Summary updated

Abstract

There exists distrust in simulation validation all the time. A quantitative approach is proposed to obtain measurable, comparable judgments of simulation correctness. The commonality between machine learning and simulation model validation is analyzed. We focus on the idea of applying cross validation in the area of simulation validation. Based on cross validation, a strategy is proposed to predict the fit of a simulation model to a validation set. Scaling factor is then introduced into the approach to improve its efficiency. The approach is applied in a simulation system to verify the usefulness of the approach proposed. The result shows it is convinient to get an effective estimate of correctness of simulation models with the method.
BibTeX
@misc{lin2018application,
  author = {Qi Lin and Yong Chen},
  title = {{Application of Machine Learning Method in Simulation Model Validation}},
  year = {2018},
  doi = {10.2991/eame-18.2018.48},
}
Figure