TL;DR
This paper tackles the challenge of accurately identifying purchasable items in voice shopping commands, a crucial aspect for user trust and experience. They introduce a novel framework that combines semantic search with a "twin product classifier" to distinguish between items that can be bought and those that cannot, significantly outperforming existing methods.
What Problem Does This Paper Address?
Voice-controlled assistants like Alexa and Google Assistant rely on Spoken Language Understanding (SLU) to interpret user commands. SLU typically involves two main tasks: intent classification (what the user wants to do) and slot filling (extracting key information like product names). A critical challenge arises in "shopping" intents, where distinguishing between a purchasable item and a similar but non-purchasable entity (e.g., "buy batteries" vs. "how much is Ronaldo worth?") is difficult. Existing SLU models often struggle with this, leading to "false accepts" (thinking a non-purchasable item is buyable) or "false rejects" (failing to recognize a purchasable item). This ambiguity can erode user trust if they accidentally purchase something or become frustrated if they can't buy what they intend. The problem is exacerbated by the sheer scale of product catalogs, making exhaustive searches computationally prohibitive.
Methods
The proposed solution integrates a retrieval module with a twin product classifier.
- Retrieval Module: Instead of searching the entire product catalog for every detected item name (an "ItemName candidate"), this module uses semantic search to efficiently retrieve a small set of the most similar products from the catalog. This drastically reduces the search space.
- Twin Product Classifier: This component, inspired by Siamese networks, takes two inputs: the original user utterance and the set of retrieved product matches. It's trained using a contrastive loss function. The goal isn't to classify the inputs directly but to learn to differentiate between pairs that represent a "match" (the ItemName candidate is indeed purchasable) and pairs that don't. This involves learning to "pull together" embeddings of matching utterance-product pairs and "push apart" embeddings of non-matching pairs. The classifier then provides a signal to adjust the confidence scores of the initial SLU intent/slot predictions.
Datasets
The paper uses a dataset derived from live traffic, annotated over a year.
- Training Data: Contains positive and negative examples of utterances with ItemName candidates. Positive examples are utterances with a "Shopping" intent, while negative examples are those without. The dataset has more negative than positive examples.
- Product Catalog: Instead of the full product catalog, they use the top one million search queries as a representative subset. This catalog is used for the retrieval module.
- Validation/Test Data: The dataset is in German and anonymized.
Key Insights and Ideas
- Semantic Search is Key for Efficiency: Exhaustively searching billions of products is impractical. Semantic search, combined with a retrieval module, efficiently narrows down potential matches, making the problem tractable.
- Twin Networks Excel at Discrimination: The core idea of learning to differentiate between similar but distinct concepts (purchasable vs. non-purchasable) is well-suited for twin network architectures and contrastive learning.
- Context Matters: The Whole Utterance is Informative: Surprisingly, including the entire utterance (not just the ItemName candidate) in the twin network's comparison significantly improved performance, suggesting that the surrounding words provide crucial context for disambiguation.
- More Matches are Better (Up to a Point): Retrieving the top five matching products yielded better results than just the first match, as it provided a more robust characterization of the ItemName candidate. However, using only one match can lead to more false accepts.
- Contrastive Loss is Superior for This Task: Among the tested loss functions, the online contrastive loss proved most effective for this specific problem of distinguishing purchasable items.
Key Results
The proposed Twin Product Classifier significantly improves performance over a baseline IC-SF system:
- Accuracy: +81.58% improvement.
- F1 Score: +42.27% improvement.
- Precision: +97.67% improvement.
- Recall: -11.55% decrease. (This decrease is attributed to the dataset's imbalance, with more false accepts than false rejects.)
Other notable results:
- Using the top five retrieved products instead of just one improved accuracy by +14.95%.
- Including the whole utterance in the twin network improved accuracy by +8.53% and F1 by +9.72%.
- Euclidean distance slightly outperformed cosine similarity and Manhattan distance for the twin network's comparison.
Limitations and Future Work
- Dataset Imbalance: The dataset contains more false accept samples than false reject samples, making it challenging to improve recall without introducing more false accepts. Creating datasets with a higher proportion of false rejects is noted as difficult due to annotation limitations.
- Future Work: The authors suggest further exploration of different loss functions and distance metrics. The integration of the Twin Product Classifier into a larger SLU system is presented as a flexible component that can be added to existing architectures.
Summary updated