Abstract
In safety-critical environments such as aircraft cockpits, automated systems must reliably interpret human context to provide life-saving assistance. This paper introduces a neuro-symbolic dialogue system that combines the flexibility of Large Language Models (LLMs) for Natural Language Understanding (NLU) with the rigorous reasoning of symbolic planning. By utilizing Fully Observable Non-Deterministic (FOND) and Hierarchical Task Network (HTN) planning, the system actively engages pilots to resolve emergencies. Evaluation across multiple scenarios shows the system achieves a 100% success rate, significantly outperforming pure LLM-based chatbots in complex, iterative reasoning tasks.
Introduction
In an emergency landing of an ultralight aircraft, context is paramount. Automated systems can monitor "passive" data like engine temperature, but they often lack the "active" context known only to the pilot, such as the smell of smoke or physical symptoms like dizziness.
Current conversational AI, while fluent, suffers from a "reliability gap." LLMs are prone to hallucinations and struggle with the sequential reasoning required to navigate a checklist-driven emergency. Conversely, traditional rule-based systems are often too rigid for natural human speech. This research proposes a hybrid neuro-symbolic approach to create a system that is both linguistically flexible and mathematically sound.
System Architecture
The proposed architecture splits the workload between three specialized components to ensure both understanding and reliability.
- Natural Language Understanding (NLU): The system uses a DIET (Dual Intent and Entity Transformer) architecture based on BERT to extract "intents" and "entities" from raw speech. This maps messy human input into structured variables.
- Dialogue Management: Instead of an LLM deciding the next response, the system employs FOND planning. It treats the conversation as a roadmap where the goal is to fill specific information "slots" (e.g., smoke color). This ensures the dialogue remains goal-directed.
- Instruction Generation: Once the context is established, the system uses HTN planning to generate a step-by-step emergency procedure tailored to the specific situation.
Data Augmentation and Diversity
A significant hurdle in safety-critical AI is the lack of real-world emergency data. To address this, the authors solicited data from five licensed pilots to understand natural verbalization patterns. They then used GPT-4-turbo-preview to generate realistic synthetic utterances to augment the training set.
To validate this approach, the researchers compared the synthetic data against human data using "Remote Clique" and "Chamfer Distance" metrics. These metrics measure the diversity and spread of the utterances in a high-dimensional space. The results demonstrated that the LLM-augmented training data was just as diverse as real human speech, making it a viable tool for training specialized NLU models without compromising quality.
Key Insights and Results
The system was benchmarked against GPT-3.5 and GPT-4 in three scenarios: landing gear malfunction, general health emergency, and fire emergency.
- Iterative Reasoning: The study confirmed that while LLMs are proficient at extracting information from a single sentence, they struggle with iterative reasoning. In the complex "Fire Emergency" scenario, GPT-3.5 succeeded only 14% of the time, and GPT-4 succeeded 42%. In contrast, the proposed neuro-symbolic system achieved a 100% success rate.
- Transparency and Safety: Because the dialogue is driven by a symbolic FOND plan, the system’s logic is transparent. This "active inference" allows the system to ground its world model in the pilot's actual perception, ensuring both the AI and the human are working from the same facts.
Conclusion
While the system excels in structured emergencies, future work is needed to handle mid-conversation goal changes (e.g., a medical issue arising during a fire). The authors plan to incorporate Abstract Meaning Representation (AMR) to improve the understanding of complex sentences and test the system in high-fidelity simulators to evaluate performance under the stress and noise of a real cockpit environment.
Summary updated