Neural-Bayesian Program Learning for Few-shot Dialogue Intent Parsing

Mengze Hong, Di Jiang, Yuanfeng Song, Chen Jason ZhangarXiv 2024 · arXiv

TL;DR

The authors introduce DI-Parser (Dialogue-Intent Parser), a hybrid Neural-Bayesian Program Learning model designed to tackle dialogue intent parsing in data-scarce environments. By combining the structural flexibility of Bayesian networks with the strong representation capabilities of neural networks, DI-Parser successfully extracts customer intents using only a few labeled examples. Evaluated on a massive real-world industrial dataset, it consistently outperforms state-of-the-art deep learning models like Transformers.

What Problem Does This Paper Address?

In customer service, understanding the "intent" behind a user's utterance (e.g., distinguishing a "price inquiry" from a "complaint") is vital for automation and analytics. However, modern deep learning approaches are notoriously data-hungry. Building an intent parser for a new domain typically requires massive, manually labeled datasets and tedious feature engineering.

Furthermore, traditional models are rigidly bound to a predefined intent taxonomy. If a customer introduces a novel problem, standard models cannot dynamically adapt or discover this new intent without being completely retrained on new data. The authors address this bottleneck by designing a system that can learn to parse intents efficiently from sparse data (few-shot learning) while retaining the ability to discover new intents organically.

Methods

DI-Parser is built on the principles of Neural-Bayesian Program Learning (NBPL). Instead of treating intent parsing as a pure classification problem, it models dialogue as a probabilistic generative process composed of four distinct, transferable components:

  1. Intent ($z_j$): Modeled as a multinomial distribution over words. To keep the model compact, a similarity metric checks new intents against existing ones to filter out redundancies.
  2. Intent Distribution ($\theta$): Captures long-range dependencies within a dialogue (e.g., a sales dialogue naturally features different intent frequencies than a technical support dialogue) using a Dirichlet process prior.
  3. Intent Transition ($M$): A 2-dimensional matrix capturing local turn-by-turn dynamics (e.g., a "greeting" is usually followed by a "request").
  4. Neural Embedding ($e$): Utterance representations generated by a neural network and mathematically modeled via a Gaussian distribution.
Figure 1: Graphical model of our proposed DI-Parser, whose learning process is initialized by learning to learn.

The "Learning to Learn" Framework

The core innovation lies in how these components interact. DI-Parser uses a small set of human-annotated exemplars to initialize hierarchical priors (the "learning to learn" phase). It then transfers these learned structures to guide unsupervised inference on large volumes of unlabeled data using Gibbs sampling. This allows the model to dynamically discover new intents while anchoring its understanding on the provided few-shot examples.

Datasets

The model was evaluated on a massive, real-world industrial dataset:

  • Source: 952,749 customer service dialogues from the financial domain, transcribed using audio-to-text methods.
  • Split: 80% of the data was used to compile the base Intent Set, and 20% was reserved for testing.
  • Few-Shot Exemplars: To simulate a data-hungry environment, a small subset of 1,000 dialogues was annotated by humans via Amazon Mechanical Turk (AMT) to serve as the few-shot training foundation.

Key Insights and Ideas

  • Decomposition Enables Transfer: By breaking dialogue down into distinct structural components (intents, transitions, distributions), the model makes domain adaptation much easier. Turn-taking behavior (transitions) and word meanings (embeddings) transfer naturally across domains even if the exact intent frequencies change.
  • Bayes Meets Deep Learning: Pure deep learning struggles with sample efficiency, while pure Bayesian models struggle with raw semantic representation. Combining them allows Gaussian-distributed neural embeddings to handle semantic nuances while the Bayesian framework handles structural logic.
  • Dynamic Taxonomy: Unlike rigid classifiers, DI-Parser's use of unsupervised topic modeling alongside a word-distribution similarity threshold allows it to expand its intent vocabulary dynamically without human intervention.

Key Results

DI-Parser was benchmarked against several standard text classification and sequence models.

Extracted table from page 6 of Neural-Bayesian Program Learning for Few-shot Dialogue Intent Parsing

As shown in the results table, DI-Parser achieved an accuracy of 79.6%, outperforming all baselines. Notably, it achieved a 1.7% improvement over the state-of-the-art Transformer model under the same few-shot constraints, bringing the system significantly closer to human inter-annotator agreement levels.

Limitations and Future Work

While the paper demonstrates strong results, a few limitations are worth noting:

  • Inference Latency: The framework relies on Gibbs sampling for parsing and inference. While highly accurate for probabilistic models, sampling methods are computationally heavier and slower than the simple forward pass of standard neural networks, which could challenge real-time industrial deployment.
  • Dependence on Initial Exemplars: The performance of the "learning to learn" framework is heavily dependent on the quality and diversity of the initial 1,000 human-annotated dialogues.

For future work, the authors suggest scaling this Neural-Bayesian approach to broader text-mining applications and exploring tighter integrations with modern Large Language Models (LLMs) to enhance the initial embedding space.

Summary updated

Abstract

With the growing importance of customer service in contemporary business, recognizing the intents behind service dialogues has become essential for the strategic success of enterprises. However, the nature of dialogue data varies significantly across different scenarios, and implementing an intent parser for a specific domain often involves tedious feature engineering and a heavy workload of data labeling. In this paper, we propose a novel Neural-Bayesian Program Learning model named Dialogue-Intent Parser (DI-Parser), which specializes in intent parsing under data-hungry settings and offers promising performance improvements. DI-Parser effectively utilizes data from multiple sources in a "Learning to Learn" manner and harnesses the "wisdom of the crowd" through few-shot learning capabilities on human-annotated datasets. Experimental results demonstrate that DI-Parser outperforms state-of-the-art deep learning models and offers practical advantages for industrial-scale applications.
BibTeX
@misc{hong2024neural,
  author = {Mengze Hong and Di Jiang and Yuanfeng Song and Chen Jason Zhang},
  title = {{Neural-Bayesian Program Learning for Few-shot Dialogue Intent Parsing}},
  year = {2024},
  eprint = {2410.06190},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.06190},
}
Figure