Knowledge Graph - Deep Learning: A Case Study in Question Answering in Aviation Safety Domain

Ankush Agarwal, Raj Gite, Shreya Laddha, Pushpak Bhattacharyya, Satyanarayan Kar, Asif Ekbal, Prabhjit Thind, Rajesh Zele, Ravi ShankararXiv 2022 · arXiv

TL;DR

This paper introduces a hybrid Question Answering (QA) system for aviation safety that combines the structured reasoning of Knowledge Graphs (KG) with the broad coverage of Deep Learning (DL) models like BERT and GPT-3. By integrating these two paradigms, the researchers achieved a significant performance boost, with their combined system outperforming standalone GPT-3 by 9.3% and BERT-QA by over 40% in answer accuracy.

What Problem Does This Paper Address?

The aviation industry generates a staggering volume of technical documentation, including National Transportation Safety Board (NTSB) accident reports and Federal Aviation Administration (FAA) regulatory directives. For safety professionals, investigating trends or specific accident causes across thousands of multi-page documents is a "needle in a haystack" problem.

Existing solutions generally fall into two camps, both with flaws:

  1. Knowledge Graphs: These are excellent at representing structured relationships (e.g., Accident X was caused by Component Y), but they are expensive to build and often lack the "expressive" nuances of natural language found in raw text.
  2. Deep Learning Models: Large language models (LLMs) have high coverage and can process unstructured text, but they often lack specific domain expertise and struggle with "multi-hop" reasoning—questions that require connecting disparate pieces of information across a document.

Methods

The authors propose a dual-module architecture that processes natural language queries in parallel:

1. The Aviation Knowledge Graph (AKG)

The team constructed a domain-specific KG using 4,000 NTSB reports. They mapped these reports to the ADREP Taxonomy (a global standard for aviation accident reporting).

  • Ontology Creation: They used a mix of automated techniques (NER, TF-IDF, and BERT embeddings) and manual domain-expert validation to define classes like "Aircraft Issue" or "Environmental State."
  • KGQA Module: To query the graph, they built a pipeline that translates natural language into SPARQL queries. This involves classifying the question type (e.g., Boolean vs. List) and extracting entities to match the graph's structure.
Figure 1: Aviation Knowledge Graph Construction Process from NTSB reports

2. The Deep Learning QA (DLQA) Module

This module uses a standard "Retriever-Reader" pipeline:

  • Retriever: Uses Sentence-BERT or GPT-3 'ada' to find the top-k most relevant text passages from the document corpus.
  • Reader: Extracts (BERT) or generates (GPT-3 'curie') an answer from those passages.

3. The Hybrid System

The final system merges the outputs. If the KGQA module finds a structured answer, it is prioritized; otherwise, the system relies on the DLQA module's ability to extract information from unstructured text.

Figure 2: Knowledge Graph guided Deep Learning based Question Answering system

Datasets

  • NTSB Reports: 4,000 reports from 1962 to 2015 were used to build the KG.
  • ADREP Taxonomy: Used as the foundational schema for the ontology.
  • Curated Test Set: Because standard QA benchmarks don't cover aviation jargon, the authors handcrafted a test set of 150 query-answer-passage instances based on 50 NTSB reports from the year 2002.

Key Insights and Ideas

  • Complementary Strengths: The "aha moment" is the empirical proof of synergy. The KG excels at multi-hop questions (e.g., "Which accidents involved aircraft operated by X and manufactured by Y?") where LLMs often fail. Conversely, the DL module handles descriptive, "long-tail" information that wasn't captured during the KG's construction.
  • Domain Jargon is a Barrier: Standard BERT models performed poorly (near baseline) because they weren't fine-tuned on aviation-specific technical language.
  • Semantic vs. Exact Match: The authors argue that "Semantic Accuracy" is a better metric for safety experts than "Exact Match," as it captures whether the system understood the intent of the safety query, even if the phrasing differs from the source text.

Key Results

The hybrid approach showed clear dominance across multiple metrics. The KGQA + GPT3-QA model emerged as the top performer.

Evaluation Results Table
  • Answer Accuracy: The hybrid KGQA + GPT3-QA achieved 85.3% Semantic Accuracy, compared to 76% for standalone GPT-3 and 56% for standalone KGQA.
  • Passage Retrieval: The hybrid system reached 86.6% Semantic Accuracy in finding the right document sections, significantly aiding human investigators who need to verify answers.
  • KG Scale: The resulting Aviation KG is substantial, containing nearly 98,000 axioms and over 8,800 individuals (see Figure 3/Table 1 in the paper for full stats).

Limitations and Future Work

  • KG Scalability: Constructing the ontology still requires significant manual effort from domain experts, making it difficult to scale to every possible aviation sub-domain.
  • NL2SPARQL Errors: The rule-based pipeline for converting natural language to SPARQL is prone to errors if the user's phrasing is highly unconventional.
  • Future Directions: The authors plan to expand the KG by incorporating ASRS (close-call) reports and FAA Airworthiness Directives. They also aim to improve the re-ranking mechanism to better decide when to trust the KG versus the DL module.

Summary updated

Abstract

In the commercial aviation domain, there are a large number of documents, like, accident reports (NTSB, ASRS) and regulatory directives (ADs). There is a need for a system to access these diverse repositories efficiently in order to service needs in the aviation industry, like maintenance, compliance, and safety. In this paper, we propose a Knowledge Graph (KG) guided Deep Learning (DL) based Question Answering (QA) system for aviation safety. We construct a Knowledge Graph from Aircraft Accident reports and contribute this resource to the community of researchers. The efficacy of this resource is tested and proved by the aforesaid QA system. Natural Language Queries constructed from the documents mentioned above are converted into SPARQL (the interface language of the RDF graph database) queries and answered. On the DL side, we have two different QA models: (i) BERT QA which is a pipeline of Passage Retrieval (Sentence-BERT based) and Question Answering (BERT based), and (ii) the recently released GPT-3. We evaluate our system on a set of queries created from the accident reports. Our combined QA system achieves 9.3% increase in accuracy over GPT-3 and 40.3% increase over BERT QA. Thus, we infer that KG-DL performs better than either singly.
BibTeX
@misc{agarwal2022knowledge,
  author = {Ankush Agarwal and Raj Gite and Shreya Laddha and Pushpak Bhattacharyya and Satyanarayan Kar and Asif Ekbal and Prabhjit Thind and Rajesh Zele and Ravi Shankar},
  title = {{Knowledge Graph - Deep Learning: A Case Study in Question Answering in Aviation Safety Domain}},
  year = {2022},
  eprint = {2205.15952},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.15952},
}
Figure