TL;DR
This paper tackles the challenge of improving e-commerce search query spell correction, especially for the prevalent phonetic and compounding errors that traditional methods struggle with. They introduce novel synthetic data generation techniques and a curriculum learning strategy for Transformer models, achieving significant improvements in accuracy and user experience metrics.
What Problem Does This Paper Address?
In e-commerce, misspelled search queries are a major hurdle, leading to irrelevant search results and a poor user experience. While typographic errors (like typos) are common, a significant portion of errors stem from users not knowing the correct spelling and instead typing words phonetically or incorrectly compounding them. Traditional spell correction methods, often relying on edit distance or simple noisy channel models, are computationally expensive, struggle with contextual errors, and fail to capture the nuances of phonetic and compounding mistakes. Furthermore, obtaining large, high-quality labeled datasets for training advanced models like Transformers is a significant bottleneck due to the cost and time involved in human annotation.
Methods
The authors propose a multi-pronged approach to address these challenges:
- Novel Synthetic Data Generation: They go beyond standard edit-distance based error generation.
- Phonetic Errors: They leverage machine translation techniques, specifically training character-level sequence-to-sequence models to transliterate English words to Hindi (and vice-versa). This is then augmented with common Hindi spelling mistakes and back-transliterated to English to create synthetic phonetic errors. They also use traditional methods like Soundex and Metaphone.
- Edit/Phonetic + Compounding Errors: They combine phonetic and edit-distance errors with compounding errors, mimicking real-world complex mistakes.
- Query Chains: They utilize sequences of user queries where a user corrects their own spelling in a subsequent query.
- Transformer-based Model: They employ a state-of-the-art Transformer architecture, known for its effectiveness in sequence-to-sequence tasks.
- Curriculum Learning: To handle the complexity of different error types, they implement a curriculum learning strategy. This involves first training the model on simpler, noisier synthetic data, and then fine-tuning it on more complex error types or cleaner, weakly supervised data.
- Weakly Supervised Data: They incorporate user click feedback (where users click on system-corrected queries) as a source of noisy but valuable training data.
Datasets
- Training Data:
- Synthetic Data: Approximately 270 million synthetic query pairs were generated using the novel techniques described above, covering edit, phonetic, and compounding errors.
- Weakly Supervised Data: Collected from user click feedback after deploying an initial version of the spell correction model.
- Clean Queries: A seed set of 300K clean queries from e-commerce search logs, selected based on high volume and click-through rates.
- Evaluation Data:
- Human Labeled Data: 72K human-labeled query pairs, stratified into "head" (high frequency) and "tail" (low frequency) queries, used to create "Regression" (mostly clean) and "Improvement" (mostly misspelled) datasets.
Key Insights and Ideas
- Phonetic errors are a major challenge: The paper highlights that phonetic errors, influenced by regional accents and colloquialisms, are the most prevalent type of error (37%) and are particularly difficult for traditional methods.
- Synthetic data is crucial for low-resource scenarios: The reliance on expensive human-labeled data is a major bottleneck. Their novel synthetic data generation, especially for phonetic errors, effectively bypasses this limitation.
- Context matters for spell correction: Simple word-level corrections are insufficient. The Transformer model's ability to consider context is key to its success.
- Curriculum learning is effective for complex tasks: Gradually exposing the model to increasingly difficult error types or data sources leads to better performance and avoids catastrophic forgetting of simpler errors.
- Weak supervision provides continuous learning: User feedback loops are vital for adapting to evolving user behavior and new error patterns.
Key Results
- Offline Evaluation: The proposed model, ReparoS-C2, achieved state-of-the-art performance, outperforming statistical models (GoogleSWBS) and other neural approaches like NeuSpell-BERT. For instance, ReparoS-C2 achieved 46.09% accuracy on the "Improvement" set and 91.62% on the "Regression" set, significantly better than baselines.
- Online Evaluation (A/B Tests): ReparoS-C2, deployed in production, demonstrated significant improvements in key e-commerce metrics:
- Reduced "click backs" (users explicitly asking for results for their original query) and "Null Searches" (searches with no results).
- Increased "PageCTR" (Page Click-Through Rate) and "CartAdd" (products added to cart).
- A 0.01% improvement in PageCTR was considered statistically significant at their scale.
Limitations and Future Work
- Latency of Deeper Models: While deeper Transformer models offer higher accuracy, their latency on CPUs (the typical serving infrastructure) is a significant concern for real-time e-commerce search. The authors chose a simpler 1-layer encoder/decoder for ReparoS-C2 to balance accuracy and latency.
- Future Work: The authors plan to focus on:
- Model pruning, quantization, and knowledge distillation to reduce latency.
- Continued exploration of continual learning for ongoing adaptation.
Summary updated