Supervised Domain Enablement Attention for Personalized Domain Classification

Joo-Kyung Kim, Young-Bum Kim

Introduction

Modern Intelligent Personal Digital Assistants (IPDAs), such as Amazon Alexa, must accurately route user requests to thousands of third-party "domains" or skills. To improve routing accuracy, these systems leverage personalized "enabled domain information"—the specific set of skills a user has activated.

However, traditional attention mechanisms typically rely on the softmax function to weight these enabled domains. Softmax forces attention weights to sum to 1.0, which introduces two significant issues: the irrelevance problem, where the model is forced to select a domain even if none are relevant, and the multi-relevance problem, where the model struggles to assign high confidence to multiple relevant domains simultaneously.

Proposed Methodology

To address these limitations, researchers at Amazon Alexa proposed a novel architecture that replaces standard softmax attention with a supervised sigmoid-based mechanism. The architecture consists of three primary innovations:

  1. Sigmoid Attention: By using the logistic sigmoid function instead of softmax, the model treats each domain's relevance as an independent binary decision. This allows weights to range from 0 to 1 independently, enabling the model to ignore all domains or focus on several at once.
  2. Supervised Attention: An auxiliary loss function ($L_a$) is introduced to explicitly penalize the model if it fails to assign high attention weights to the ground-truth domain when it is present in the user's enabled list.
  3. Self-Distillation: To capture nuanced relationships between similar domains (e.g., two different weather skills), the model uses self-distillation ($L_d$). It treats its own high-performing outputs from previous training epochs as "soft targets," preventing the supervised attention from becoming overly narrow.
The proposed model architecture featuring BiLSTM for utterance encoding and supervised sigmoid attention for domain enablement.

The final utterance vector from a BiLSTM is concatenated with a weighted sum of enabled domain embeddings and passed through a feed-forward network for final classification.

Experimental Results

The model was evaluated using a massive dataset of 4.4 million utterances from Amazon Alexa logs, covering approximately 1,000 domains. Performance was measured across two test sets: a Biased set (90% ground-truth inclusion in enabled domains) and an Unbiased set (70% inclusion), the latter reflecting more realistic "live" traffic conditions.

As shown in the results table below, the full proposed model (Model 4: sgmd, spvs, sdst) consistently outperformed standard softmax-based approaches (Model 1: sfm). On the unbiased test set, the proposed method improved Top1 accuracy from 90.65% to 91.33%.

Performance comparison across different attention methods on biased and unbiased ground-truth inclusion test sets.

Qualitative analysis further demonstrated that the sigmoid attention mechanism effectively ignores "distractor" domains that often confuse softmax models. For example, when a user says "find my phone," the proposed model correctly ignores irrelevant enabled skills like "The Name Game."

Conclusion and Future Work

The "Supervised Enablement Attention" mechanism significantly improves domain classification by relaxing the constraints of softmax and providing explicit supervision for attention weights. While the model shows a slight tendency to become over-reliant on enablement signals, this is mitigated through random sampling during training. Future research will focus on scaling this approach to tens of thousands of domains and improving performance for "long-tail" skills with limited training data.

Summary updated

Abstract

In large-scale domain classification for natural language understanding, leveraging each user's domain enablement information, which refers to the preferred or authenticated domains by the user, with attention mechanism has been shown to improve the overall domain classification performance. In this paper, we propose a supervised enablement attention mechanism, which utilizes sigmoid activation for the attention weighting so that the attention can be computed with more expressive power without the weight sum constraint of softmax attention. The attention weights are explicitly encouraged to be similar to the corresponding elements of the ground-truth's one-hot vector by supervised attention, and the attention information of the other enabled domains is leveraged through self-distillation. By evaluating on the actual utterances from a large-scale IPDA, we show that our approach significantly improves domain classification performance.
BibTeX
@inproceedings{kim2018supervised,
  author = {Joo-Kyung Kim and Young-Bum Kim},
  title = {{Supervised Domain Enablement Attention for Personalized Domain Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1106},
}
Figure