TL;DR
This paper introduces the Entity Transformer (ET), a novel sequence-to-sequence model designed to extract multiple product names from e-commerce voice commands, a task that current systems struggle with. The ET model, leveraging pre-trained language models and a Conditional Random Field decoder, significantly improves the efficiency and user experience of voice shopping by enabling the recognition of up to 10 items in a single utterance.
What Problem Does This Paper Address?
Modern e-commerce voice assistants, like Amazon Alexa, are limited to recognizing only a single product name per utterance. This is a significant bottleneck for users who want to add multiple items to their shopping cart efficiently. The core challenge lies in accurately identifying and segmenting multiple product names within a single, often punctuation-lacking, voice command. Traditional Named Entity Recognition (NER) models often falter in this "Multiple Product Name Entity Recognition" (MPNER) task due to the inherent ambiguity and lack of clear delimiters in spoken language.
Methods
The authors propose an Entity Transformer (ET) architecture, a sequence-to-sequence model inspired by encoder-decoder frameworks. The core components are:
- Transformer Encoder: This part takes the input sequence (the voice command text) and encodes it into a sequence of continuous representations. It utilizes stacked transformer layers with relative positional attention, multi-headed attention, and feed-forward layers.
- Conditional Random Field (CRF) Decoder: Following the encoder, a CRF layer is employed to jointly model the sequence of predicted entity tag labels. This helps in making more informed tagging decisions based on the context of the entire sequence. The model uses the BILOU tagging scheme.
- Feature Engineering: The ET model incorporates both dense and sparse features.
- Dense Features: These are derived from pre-trained language models like conveRT, BERT, and RoBERTa.
- Sparse Features: These include one-hot token-level encodings and multi-hot character n-gram features (up to n=4). These sparse features are processed by a feed-forward layer before being concatenated with the dense features for the transformer encoder.
Datasets
The authors created a new MPNER dataset for their evaluation.
- Size: Approximately 1 million product name entities extracted from voice order shopping utterances.
- Split: A 500k training subset and a 65k test subset.
- Generation: The dataset was generated by augmenting 70 seed voice utterances with synonyms for popular products across various departments. A key characteristic is the random number of products (1-10) per utterance, simulating real-world shopping scenarios. The test set includes 5k unseen products.
Key Insights and Ideas
- The "No Punctuation" Problem: The lack of punctuation in ASR output is a major hurdle for MPNER, making it difficult to distinguish between adjacent product names.
- Pre-trained Models Aren't Enough: While powerful, general-purpose pre-trained language models like BERT don't perfectly generalize to the nuances of MPNER without task-specific adaptation.
- Combining Strengths: The best performance was achieved by combining the power of pre-trained dense embeddings (specifically conveRT) with handcrafted sparse features (word and character n-grams) within the ET architecture. This suggests a hybrid approach is highly effective for this task.
- CRF for Sequence Tagging: The use of a CRF decoder is crucial for jointly modeling the sequence of entity tags, leading to more coherent and accurate predictions.
Key Results
- The best performing ET model (conveRT + ngram + ET) achieved a 12% performance improvement on their test set compared to a non-neural baseline.
- This model also outperformed BERT with ET, demonstrating the effectiveness of their proposed architecture.
- The ET model can recognize up to 10 items in a single utterance.
- The average inference time for the ET model is 80 ms per voice reorder utterance, making it efficient for real-time applications.
- Training times vary significantly with hardware, with a model trained on a single V100 GPU taking approximately 7 hours for 100 epochs.
Limitations and Future Work
- Generalization to Unseen Products: While the test set includes unseen products, the authors plan to investigate customized pre-trained models specifically for the MPNER task to further improve generalization.
- Data Augmentation: The current data augmentation relies on synonyms. Future work could explore more sophisticated augmentation techniques to create even more robust training data.
- Model Complexity: While effective, the ET model is a complex architecture. Further research could explore more lightweight variants for resource-constrained environments.
Summary updated