TL;DR
This paper introduces Graph Integrated Language Transformers (GaLT), a novel approach to predict the next action in complex phone calls by integrating graph structures with language transformers. This method significantly improves performance and robustness compared to existing dialogue management systems, particularly in noisy, real-world scenarios, by reducing reliance on traditional NLU pipelines and handling conversational grounding issues.
What Problem Does This Paper Address?
Building and maintaining production-quality conversational AI systems, especially for complex tasks like outbound call automation, is a significant challenge. Current systems often rely on intricate dialogue managers with multiple components (NLU, state tracking, action prediction, response generation). This pipeline approach introduces complexity, increases processing time, and can lead to errors due to additive noise. A critical component is "next action prediction," which determines the system's response. Existing methods often depend on Slot-Filling (SF) and Intent-Classification (IC) techniques, which can be unstable and require external knowledge sources. Furthermore, these systems struggle with "grounding" – handling situations where user input deviates from expected patterns, leading to misunderstandings. While end-to-end LLMs show promise, they can hallucinate and underperform in domain-specific conversations requiring structured knowledge. This paper aims to address these limitations by developing a more robust and efficient next action prediction system.
Methods
The core innovation is the Graph Integrated Language Transformer (GaLT). This approach combines the power of language transformers (like BERT, DistilBERT) with graph-based methods to predict the next action.
Key aspects of the method include:
- Graph Integration: Instead of relying solely on NLU pipelines, GaLT incorporates a graph component. This component captures the co-occurrence of actions and human utterances. Two specific implementations are explored:
- GNN-LT: Uses Graph Neural Networks (GNNs) to explicitly encode the graph of actions and their relationships.
- GaLT: Employs a graph embedding layer to learn action history co-occurrences without explicitly encoding graph constraints. This is the primary model evaluated.
- Transformer Backbone: Language transformers provide the necessary language understanding capabilities.
- Reduced Dependency: The model aims to predict the next action as a single task, removing the need for separate SF/IC components and external knowledge sources.
- Overcoming Token Limits: By focusing on action history as graph nodes rather than full dialogue history, the model circumvents the token limits of transformers, improving efficiency.
- Fusion Layer: Features from the language transformer and the graph component are fused using a pairwise dot product before being fed into a fully connected layer for final prediction.
Datasets
The research utilizes conversational data that follows a Standard Operating Procedure (SOP). This data was collected from phone calls between humans and AI systems, guided by human experts or a dialogue manager system that implements a human-defined SOP.
- Training Data: Initially, approximately 1 million dialogue turns were collected. This was then filtered to around 600,000 turns by removing undesirable or obsolete actions and filler actions. The data was split into 80% for training, 10% for validation, and 10% for testing.
- Evaluation Data: The evaluation was conducted on this preprocessed dataset. The paper also mentions evaluating a prompting-only approach using Llama 2 on the same dataset.
Key Insights and Ideas
- Graph Structure is Key for Robustness: Integrating graph information about action sequences significantly improves next action prediction, especially in handling conversational grounding and unexpected user inputs.
- Decoupling from NLU Pipelines: The proposed GaLT model demonstrates that it's possible to achieve high performance without relying on traditional, potentially brittle, NLU components like Slot-Filling and Intent-Classification.
- Efficiency Gains: By focusing on action history as graph nodes and removing the need for extensive dialogue history, GaLT overcomes transformer token limits and achieves faster inference times.
- Data Efficiency: The GaLT model can achieve high performance with a relatively small amount of training data (as little as 60K dialogue turns), making it more practical for real-world deployment.
- Human-Centric Evaluation Matters: Beyond standard classification metrics, human-centered evaluations (product-level metrics and subjective ratings) reveal that GaLT is perceived as more robust and leads to better call completion rates.
Key Results
- Offline Classification Performance: GaLT achieved the highest F1-Weighted score (0.84) and F1-Macro score (0.75) among the evaluated models, outperforming transformer-only models, GNNs, and even prompt-based LLMs.
- Product-Level Metrics: In real-world evaluations, GaLT significantly outperformed the existing Dialogue Manager (DM) system in terms of call completion (E2E reach), especially in medium and hard difficulty scenarios. GaLT achieved 100% E2E reach in easy and medium difficulties, compared to 90% and 70% for the DM system, respectively. It also improved E2E success by +31.92%.
- Subjective Human Evaluation: Human evaluators rated GaLT higher than the DM system, finding it more robust and less prone to negative outcomes. While t-tests for human ratings across difficulty levels were not statistically significant, the trend favored GaLT.
Limitations and Future Work
- Domain Adaptability: The proposed models, while high-performing, require re-training or fine-tuning for new applications or domains, especially if actions are added or removed.
- Interpretability: Similar to other neural models, GaLT's lack of explicit interpretability might make it difficult to understand why a specific action is predicted, potentially leading to instability.
- Evaluation Bias: The evaluation relied on human agents and employees familiar with the DM system, which could introduce bias.
- Future Work:
- Further investigation into graph injection with additional third-party Generative AI (GenAI) prompt-based models.
- Exploring how to automate calls without dependency on other components by combining graph integration with generative AI models for both information extraction and custom response generation.
Summary updated