This paper introduces a hybrid framework that blends Large Language Models (LLMs) with physics-based modeling to turn thousands of aviation accident reports into an explainable causal knowledge graph. By combining text-mined causal chains with mechanistic failure models, it enables researchers to perform "what-if" interventional analyses and quantify risk based on real-world physical conditions rather than just historical frequencies.
What Problem Does This Paper Address?
Aviation authorities maintain massive archives of accident reports, but most computational analysis of this data is limited to "sorting" reports into fixed categories. These purely statistical approaches fail to answer the most critical question for safety analysts: How much would the risk of a severe outcome have fallen if a specific contributing factor had been absent?
Existing methods struggle with two main issues:
- Lack of Mechanism: Purely data-driven models (like Bayesian networks fitted to codes) treat risk as a simple frequency. They don't understand the underlying physics—like how temperature and humidity actually cause ice to form in a carburetor.
- Selection Bias: Accident archives only record "reportable events." This creates a "collider" effect where different causes appear to compete with each other in the data, making it difficult to perform sound causal inference without expert-built structures.
Methods
The authors propose a two-layer architecture joined by a fusion rule:
- Data Layer: A local LLM (Qwen3.6-35B-A3B) processes 56,202 NTSB and FAA report narratives. It extracts temporally ordered "causal chains" (e.g., moist air $\rightarrow$ induction icing $\rightarrow$ engine power loss $\rightarrow$ forced landing) using a closed vocabulary of 57 factors.
- Physics Layer: For failure modes with physical mechanisms (like carburetor icing or fuel exhaustion), the authors attach mechanistic models. These models map continuous physical drivers (like temperature, dewpoint, and power settings) to an occurrence probability.
- Fusion & Calibration: A "leaky noisy-OR" model combines these layers. Crucially, the physics layer provides the conditional shape of the risk (how it changes with weather), while the corpus data is used to calibrate the absolute level of risk to match the observed accident population.
- Interventional Analysis: The framework supports Pearl’s "do-operator" in two ways:
- Graph-Level: Using backdoor adjustment on the learned causal network for categorical factors.
- Physics-Level: Directly transforming physical drivers (e.g., simulating the effect of applying carburetor heat) to see how the downstream risk changes.
Datasets
The study utilizes a massive corpus of 56,202 reports drawn from three public archives:
- NTSB Aviation Databases: 43,604 reports.
- FAA Accident and Incident Data System (AIDS): 12,585 reports.
- NTSB Published Investigation Reports: 13 detailed full-length reports.
For evaluation, the authors used a sample of 2,000 reports where LLM-extracted nodes were compared against investigator-flagged "cause" and "factor" codes.
Key Insights and Ideas
- Physics as a Stable Prior: While data-driven models can "drift" as reporting practices change over decades, physics is constant. The authors found that the physics-based model was more stable across different time windows than purely statistical models.
- The "Ceiling" of Predictability: By comparing their physics model to a supervised logistic regression, the authors found that their model reached an AUC of 0.61, very close to the theoretical "density ratio ceiling" of 0.64 for the given weather features. This suggests the physics model captures nearly all the available signal in the recorded weather data.
- Selection Bias (Colliders): The paper highlights that in an accident-only database, causes often have a "lift" below one. For example, "procedure not followed" might appear negatively correlated with engine failure because if a mechanical failure is present, it "explains away" the accident, making the investigator less likely to flag a procedural error.
- Interpretability over Accuracy: The authors honestly note that the physics layer adds only a small margin (0.03 AUC) to predictive accuracy. Its true value lies in interpretability—allowing an analyst to trace a risk reduction back to a specific physical intervention like "apply carburetor heat."
Key Results
- High Extraction Fidelity: The LLM-based causal chain extraction recovered 90.8% of investigator-flagged causes from the narratives.
- Icing Model Performance: The carburetor icing model successfully separated 1,205 icing accidents from 38,565 other accidents using only temperature and dewpoint.
- Counterfactual Analysis: Using abduction on four specific accidents, the model estimated a "probability of necessity" for heat omission between 0.29 and 0.50. This quantifies how likely it is that the accident would have been avoided had carburetor heat been used.
- Leading Factors: The framework identified "Fuel exhaustion or starvation" as the dominant contributor to engine failure, with a fitted strength ($p_i$) of 0.96.
Limitations and Future Work
- The "Safe Flight" Gap: Because the corpus only contains accidents, the model cannot calculate absolute per-flight risk without external "exposure" data (e.g., total flight hours in specific weather conditions).
- Human Factors: While physical failures have mechanistic models, human and organizational factors (like "inadequate maintenance") remain purely data-driven. The authors suggest that future work could integrate cognitive or organizational models.
- Model Idealization: The physics models are necessarily simplified. For instance, the carburetor icing model uses fixed cooling values ($25^\circ\text{C}$ at cruise, $32^\circ\text{C}$ at descent) which may vary across different aircraft engines.
- Investigator Bias: Part of the model's performance might reflect "investigator practice," as investigators often use the same published icing charts the authors used to build their physics model.
Summary updated