Validating Uncertain Aircraft Simulation Models Using Flight Test Data

Andrei Dorobantu, Peter Seiler, Gary Balas2013 · DOI

TL;DR

This paper introduces a novel method for validating uncertain aircraft simulation models using flight test data. By linking a statistical metric (Theil's Inequality Coefficient) to robust control theory, the authors provide a rigorous yet broadly applicable approach to ensure simulation accuracy for safety-critical flight control systems.

What Problem Does This Paper Address?

Developing reliable flight control systems for aircraft hinges on having accurate simulation models. These models are crucial for testing and validating control algorithms before they are deployed in real aircraft. However, aircraft dynamics are inherently complex and uncertain, making it challenging to create simulation models that precisely match real-world behavior across all flight regimes. Current model validation techniques often fall into categories: analytically rigorous but limited to simple linear systems, statistically broad but lacking guarantees, or based on empirical experience. The aerospace industry needs a unified, efficient, and robust method to validate these complex, uncertain, nonlinear simulation models against flight test data.

Methods

The core of the paper's contribution lies in its proposed model validation approach. It starts by defining Theil's Inequality Coefficient (TIC) as a metric to quantify the similarity between a simulation's time history and actual flight data. The authors then establish a crucial theoretical link: for simple linear time-invariant (LTI) systems with multiplicative input uncertainty, calculating the TIC is equivalent to performing a robust control analysis. This means the TIC can be bounded by robust control metrics, providing a measure of confidence in the model's accuracy based on its uncertainty description.

For nonlinear systems, this direct analytical link is lost. To overcome this, the paper extends the approach using Monte Carlo simulations. Here, the uncertain parameters of the simulation model are randomly sampled, and numerous simulations are run using flight data inputs. This generates a distribution of predicted TIC values. The validation then involves comparing the TIC calculated from actual flight data against this predicted distribution. If the flight data TIC falls within a confidence bound derived from the Monte Carlo simulations, the model is considered valid.

Datasets

The paper utilizes flight test data from the University of Minnesota's Unmanned Aerial Vehicle (UAV) platform. This platform is described as a low-cost, experimental testbed with advanced capabilities for guidance, navigation, and flight control research. The authors specifically mention using data from inner-loop control performance tests (pitch and roll angle tracking) and a set of repeated flight experiments with a step-type command for roll angle reference. No new benchmarks are introduced; rather, existing flight test data from their UAV is used for demonstration.

Key Insights and Ideas

  • Bridging the Gap: The most significant insight is the successful theoretical connection between a statistical similarity metric (TIC) and robust control theory for linear systems. This provides a rigorous foundation for using TIC in model validation.
  • Extending to Nonlinearity: The paper elegantly addresses the challenge of applying this to nonlinear systems by leveraging Monte Carlo simulations. This allows for a probabilistic assessment of model validity even when direct analytical ties are broken.
  • TIC as a Unified Metric: The authors propose TIC as a candidate for a standardized model validation metric, offering advantages over traditional methods due to its normalized scale and broad applicability.
  • Confidence in Validation: By relating model uncertainty to a quantifiable metric, the approach provides a higher degree of confidence in the validation process, which is critical for safety-critical applications.

Key Results

The paper demonstrates the proposed approach using the University of Minnesota UAV.

  • Roll Axis Validation: For the roll rate, yaw rate, and roll angle tracking, the Monte Carlo simulation TIC distributions bounded the TIC values computed from the flight data. This indicates that the simulation model accurately represents the aircraft's behavior in these axes.
  • Yaw Axis Discrepancy: In the yaw axis, the flight data TIC exceeded the Monte Carlo simulation bounds. This suggests that the model's predicted uncertainty does not fully capture the observed behavior, potentially due to unmodeled effects like turbulence. The authors suggest increasing the modeled turbulence as a potential solution.
  • Linear Example Confirmation: For a simple linear system example, the Monte Carlo simulation results closely matched the theoretical upper-bound on the TIC derived from robust control theory (0.14).

Limitations and Future Work

  • Nonlinear System Rigor: While the Monte Carlo approach extends the TIC analysis to nonlinear systems, the direct analytical ties to robust control are lost. The authors acknowledge that an upper confidence bound is not readily available in the same rigorous sense as for linear systems.
  • MIMO Systems: The rigorous analytical ties between TIC and robust control are currently proven only for Single-Input, Single-Output (SISO) LTI systems. Extending these proofs to Multiple-Input, Multiple-Output (MIMO) LTI systems is identified as a future research direction.
  • Turbulence Modeling: The yaw axis validation issue highlights the need for more accurate modeling of external disturbances like turbulence, which can significantly impact flight data and model validation.
  • Future Work: The authors suggest that continuing work on extending the TIC-robust control ties to MIMO systems and further refining the application to nonlinear systems will be beneficial. The proposed approach shows great potential for simplifying and unifying model validation in the aerospace community.

Summary updated

Abstract

Reliability validation of safety-critical flight control systems requires accurate simulation models of the aircraft dynamics. This paper proposes a simple approach to validate such models efficiently based on flight data. A statistical model validation analysis is shown to be equivalent to a robust control analysis for simple linear systems. The analysis is extended to nonlinear systems in conjunction with Monte Carlo simulations to validate an uncertain aircraft simulation model. This approach is demonstrated using the University of Minnesota Unmanned Aerial Vehicle flight test platform.
BibTeX
@misc{dorobantu2013validating,
  author = {Andrei Dorobantu and Peter Seiler and Gary Balas},
  title = {{Validating Uncertain Aircraft Simulation Models Using Flight Test Data}},
  year = {2013},
  doi = {10.2514/6.2013-4984},
}
Figure