Latent Topic Attention for Domain Classification

Peisong Huang, Peijie Huang, Wencheng Ai, Jiande Ding, Jinchuan ZhangInterspeech 2019

TL;DR

This paper introduces a novel "Latent Topic Attention" mechanism for domain classification in spoken language understanding. By leveraging latent topics from the Biterm Topic Model (BTM) to guide attention, the approach significantly improves accuracy, especially when training data is limited, outperforming standard soft and hard attention methods.

What Problem Does This Paper Address?

In spoken language understanding (SLU) systems, accurately classifying user utterances into specific domains (e.g., weather query, booking) is crucial for routing requests to the correct processing modules. While attention mechanisms have proven effective in focusing on important words, existing methods rely solely on domain tags for training. This can be problematic when training data is scarce or when test data distributions differ significantly from training data, making it difficult to capture potentially informative words. The paper addresses this limitation by proposing a new attention mechanism that incorporates unsupervised latent topic information to better identify salient words for domain classification.

Methods

The core innovation is the Latent Topic Attention mechanism, which integrates the Biterm Topic Model (BTM) with a Bidirectional Long Short-Term Memory (BiLSTM) network.

  • BiLSTM Backbone: A standard BiLSTM is used to process the input utterance, generating contextualized hidden states for each word.
  • Biterm Topic Model (BTM): BTM is an unsupervised topic modeling technique designed for short texts. It learns latent topics by analyzing co-occurring word pairs (biterms) within documents. The paper leverages BTM to estimate topic-word distributions.
  • Latent Topic Attention: Instead of solely relying on domain tags, the attention weights are guided by the latent topics. The process involves:
  • Estimating topic-word distributions using BTM.
  • Calculating a correlation score between topics and words based on their distribution.
  • Determining a "topic difference" for each word by measuring how its ranking varies across different topics.
  • Using these topic differences to compute attention weights for the BiLSTM outputs. Words with significant topic differences are assigned higher attention.
  • Output Layer: A softmax classifier is used on the attention-weighted BiLSTM output to predict the domain label.

Datasets

The primary dataset used for training and evaluation is the SMP-ECDT (Social Media Processing Evaluation of Chinese Human-Computer Dialogue Technology) benchmark corpus. This corpus is specifically designed for Chinese utterance domain classification and includes a "chit-chat" category and 30 sub-categories within "task-oriented" dialogues, totaling 31 classes. It comprises 3736 training data items and 4528 test data items.

Key Insights and Ideas

  • Unsupervised Guidance for Attention: The most significant insight is that unsupervised topic modeling (BTM) can effectively guide attention mechanisms, complementing or even surpassing supervised domain-tag-based attention, especially in low-data scenarios.
  • Topic Diversity as a Signal: The paper highlights that words exhibiting diverse rankings across different latent topics are likely to be informative. This "topic difference" serves as a powerful signal for attention.
  • Improved Robustness with Unlabeled Data: The proposed method can be effectively trained with additional unlabeled data, leading to further improvements in domain classification accuracy, demonstrating its robustness and scalability.
  • Distinctiveness of Latent Topic Attention: Analysis of attention intersection shows that the Latent Topic Attention mechanism identifies different informative words compared to standard soft and hard attention, suggesting a more nuanced understanding of utterance semantics.

Key Results

  • Overall Performance: The Latent Topic Attention model achieved an accuracy of 79.09% on the SMP-ECDT benchmark, outperforming BiLSTM (76.40%), Soft Attention (78.62%), and Hard Attention (78.25%).
  • Topic Number Selection: The model performed best with 20 topics, achieving a validation accuracy of 93.34%, indicating that a moderate number of topics is optimal.
  • Effect of Additional Unlabeled Data: When trained with 4/5 of the test data as additional unlabeled data, the Latent Topic Attention model's accuracy on a held-out 1/5 test set increased from 79.29% to 79.90%, a notable improvement.
  • Attention Intersection: For k=1, Latent Topic Attention's top word was distinct from soft and hard attention in 55.7% of utterances. For k=3, 69.2% of utterances showed intersection in top attention words, suggesting a degree of agreement on important terms.
  • Qualitative Analysis: Visualization showed that standard attention models focused on irrelevant words (e.g., "20"), while Latent Topic Attention successfully identified key informative words like "平方 (the square)" and "等于 (equals)" in a "calc domain" example.

Limitations and Future Work

  • Limitations: The paper acknowledges that when training data is limited, standard attention mechanisms might be misled by irrelevant words that coincidentally correlate with domain tags. The proposed method aims to mitigate this by using topic diversity.
  • Future Work: The paper does not explicitly detail future work directions, but the success with additional unlabeled data suggests further exploration into semi-supervised and unsupervised learning for attention mechanisms. The effectiveness of BTM for short texts also opens avenues for applying this approach to other short-text classification tasks.

Summary updated

BibTeX
@inproceedings{huang2019latent,
  author = {Peisong Huang and Peijie Huang and Wencheng Ai and Jiande Ding and Jinchuan Zhang},
  title = {{Latent Topic Attention for Domain Classification}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure