A Simulation Model Validation and Calibration Platform

Shenglin Lin, Wei Li, Xiaochao Qian, Ping Ma, Ming YangLinköping 2018 · DOI

Abstract Simulation model validation and calibration (SMVC) is a complex endeavor involving uncertainty description, extensive simulation execution, and intricate data analysis. Precise modeling requires accounting for various uncertainty factors, such as model form assumptions, solution approximations, and random variability in inputs. This paper presents a software platform developed to assist in the SMVC process under conditions of uncertainty. We introduce an unprecedented process model encompassing uncertainty description, simulation experiment design, model validation, and model calibration. The platform's design is bifurcated into structural and functional components, utilizing the "strategy" design pattern to integrate and maintain diverse SMVC algorithms. The platform's utility is demonstrated through a flight vehicle kinematic control system case study, showing significant improvements in model consistency after calibration.

Introduction

In modern engineering, high-fidelity simulation models are indispensable for predicting system behavior in fields like aerospace and power generation. However, these models are inherently limited by two types of uncertainty: aleatory (stochastic randomness) and epistemic (lack of knowledge). Quantifying these uncertainties and aligning model outputs with real-world observations—the core of SMVC—is traditionally a fragmented and tedious process. Engineers often struggle with disparate tools for data management, statistical analysis, and optimization. There is a critical need for a unified environment that manages the entire lifecycle of simulation credibility.

The HIT-MVCP Process Model

To address these challenges, researchers at the Harbin Institute of Technology developed HIT-MVCP, an integrated platform built upon a comprehensive six-step workflow. This process model ensures that uncertainty is accounted for at every stage of the simulation lifecycle.

The comprehensive process model for SMVC under uncertainty, detailing the flow from uncertainty description (A1) to optimization (A6).

As illustrated in the process model, the workflow begins with Uncertainty Description (A1) and Experiment Schemes Design (A2). These feed into Simulation Execution (A3). The resulting model outputs are compared against experimental observations during Simulation Model Validation (A4). Finally, the calibration phase utilizes Consistency Metamodeling (A5) and Uncertainty Parameters Optimization (A6) to refine the model.

Platform Architecture and Design

The HIT-MVCP platform is designed with modularity and extensibility as core requirements. The architecture is divided into several functional subsystems, coordinated through a central data management layer.

The structural design of the HIT-MVCP platform, highlighting the modular subsystems and the human-computer interactive interface.

Key Design Pillars:

  1. Modular Subsystems: The platform separates concerns into Uncertainty, Experiment, Validation, and Calibration subsystems. This mirrors the actual engineering workflow and allows for independent updates to specific modules.
  2. The Strategy Design Pattern: To prevent the platform from becoming obsolete, the C++ implementation utilizes the "strategy pattern." This allows users to swap algorithms (e.g., changing sampling methods or optimization routines) without altering the core software architecture.
  3. Evidence-Based Validation: The platform employs an "evidence distance" approach for validation. This method is particularly effective for multi-output systems, as it converts complex data features into mathematical "evidence bodies" for robust comparison.
  4. Metamodeling for Efficiency: To reduce the computational burden of running high-fidelity simulations repeatedly during calibration, the platform constructs metamodels—efficient mathematical approximations used to find optimal parameters.

Case Study: Flight Vehicle Kinematic Control

The platform was evaluated using a flight vehicle kinematic control system simulation. The model accounted for aleatory uncertainties (atmospheric density, initial velocity) and epistemic uncertainties (initial mass, reference area).

Initially, the model showed a consistency score of 0.6581 when compared to experimental data. By utilizing the HIT-MVCP calibration loop, the epistemic parameters were optimized, resulting in a significantly improved validation score of 0.9398. This demonstrates the platform's ability to effectively "close the loop" between validation results and model refinement.

Conclusion and Future Work

HIT-MVCP represents a significant advancement in streamlining the SMVC process. By integrating validation results directly into the calibration objective function, the platform transforms validation from a passive assessment into an active driver of model accuracy. Future developments will focus on incorporating more sophisticated uncertainty theories, such as Evidence Theory and Imprecise Probability, and expanding the algorithm library to support increasingly complex, non-linear simulation models.

Summary updated

Abstract

The simulation model validation and calibration (SMVC) is a complicated work, including uncertainty description, many simulation experiments execution and complex data analysis etc. Moreover, there are many uncertainty factors such as model form assumptions and solution approximations, random variability of model inputs, etc. need to be considered when requiring a precise model. For assisting the SMVC effectively, this paper develops a software platform to validate and calibrate the simulation models when some quantities may be affected by uncertainty. First, an unprecedented process model, which includes uncertainty description, simulation experiment design, model validation and model calibration, is presented to explicate the procedures of SMVC under uncertainty. In the process model, many new model validation and calibration algorithms under uncertainty are applied based on our previous work. Second, the design of platform is divided into two parts, which consist of structure design and function design, and the software technique based on strategy pattern is introduced to integrate and maintain the SMVC algorithms. Then this platform is implemented according to its expected uses and key design requirements. Finally, application example of model validation and calibration of a flight vehicle kinematic control system is illustrated how to use the platform.
BibTeX
@inproceedings{lin2018simulation,
  author = {Shenglin Lin and Wei Li and Xiaochao Qian and Ping Ma and Ming Yang},
  title = {{A Simulation Model Validation and Calibration Platform}},
  booktitle = {Linköping},
  year = {2018},
  doi = {10.3384/ecp17142687},
}
Figure