Automatic Extraction of Reasoning Chains from Textual Reports

Gleb Sizov, Pinar ÖztürkEMNLP 2013

TL;DR

This paper introduces a novel method for automatically extracting "reasoning chains" from textual reports, such as aviation accident investigations. By representing documents as a "Text Reasoning Network" (TRN), the system can identify and visualize the logical flow of information from problem description to cause, offering a way to leverage accumulated organizational knowledge.

What Problem Does This Paper Address?

Organizations amass vast amounts of textual data documenting how problems are solved or analyzed – think medical records, accident reports, or legal documents. This expert knowledge is invaluable but often locked away, risking loss when employees leave. While search engines can find relevant documents, understanding the underlying reasoning and connections within them remains a manual, time-consuming process. This paper aims to bridge that gap by automatically extracting the "reasoning chains" that experts use to analyze problems, making this knowledge accessible and reusable.

Methods

The core of the proposed method is the Text Reasoning Network (TRN), a novel graph-based representation of textual documents. TRNs are built automatically from text using natural language processing tools. Here's how it works:

  • Text Units as Nodes: The system breaks down text into meaningful units. Instead of just words, it incorporates constituents from syntax trees (sentences, noun phrases, verb phrases) as "text nodes."
  • Structural Relations: The hierarchical structure of syntax trees is preserved using "Contains" and "PartOf" edges, reflecting how text units are nested.
  • Section Nodes: To capture document structure, "section nodes" (e.g., "Factual Information," "Analysis") are added and linked to relevant text nodes.
  • Similarity Relations: Text nodes from different sentences but of the same type (e.g., two noun phrases) are compared for similarity using WordNet-based LCH similarity and a bipartite matching algorithm. If the similarity exceeds a threshold (0.6), a "Similar" edge is added.
  • Causal Relations: A discourse parser is employed to identify causal relationships between text units, which are then represented as "Cause" edges in the TRN.

Once the TRN is constructed, reasoning chains are extracted as the shortest paths between a designated "start" node (typically a sentence from the "Summary" section) and an "end" node (usually a sentence from the "Causes" section). These paths are then post-processed and visualized.

Datasets

The authors used a dataset of aviation investigation reports from the Transportation Safety Board of Canada. These reports were downloaded as HTML documents and processed using custom Java components and ANNIE components from the GATE NLP platform. The dataset consists of reports with distinct sections like "Summary," "Factual Information," "Analysis," and "Findings as to Causes."

Key Insights and Ideas

  • Beyond Keywords: The TRN approach moves beyond simple keyword matching by capturing structural, semantic, and causal relationships within the text, providing a richer representation of the document's content and the author's reasoning.
  • Explicit Reasoning Paths: Extracting reasoning chains as paths in the TRN makes the author's logical progression explicit, allowing users to follow how initial information leads to identified causes.
  • Leveraging NLP Tools: The method effectively integrates various NLP techniques – syntactic parsing, discourse parsing, and semantic similarity – to build a comprehensive representation.
  • Visualizing Complex Information: The visualized reasoning graphs offer an intuitive way to understand complex causal relationships, even for non-experts.

Key Results

The paper demonstrates the method's ability to generate reasoning chains from aviation accident reports. The authors present several example reasoning graphs (Figures 2, 3, and 4) that successfully illustrate the connection between incident descriptions and their causes. These graphs highlight specific causal pathways, such as how focus on certain controls prevented attention to aircraft altitude, leading to an undetected sink rate. The authors note that while some chains are highly informative, others might be less so or even incoherent, suggesting areas for improvement.

Limitations and Future Work

The authors acknowledge several limitations and propose future directions:

  • Evaluation Measures: A significant limitation is the lack of established evaluation measures for reasoning chain extraction. Future work includes developing formal evaluation procedures, both intrinsic (comparing against manually constructed chains) and extrinsic (using chains in Case-Based Reasoning tasks).
  • Incoherence and Incompleteness: Some extracted chains can be incoherent (due to conflicting information or context shifts) or less informative. Future work aims to address this by exploring preferences for larger text units, recognizing negations, or adapting flow networks for path extraction.
  • TRN Representation: The TRN model could be enhanced by incorporating new types of relations.
  • Domain Adaptability: The method's adaptability to other domains and applications is highlighted, suggesting that by defining appropriate start/end nodes and constraints, it can be applied beyond aviation reports.

Summary updated

BibTeX
@inproceedings{sizov2013automatic,
  author = {Gleb Sizov and Pinar Öztürk},
  title = {{Automatic Extraction of Reasoning Chains from Textual Reports}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
}
Figure