TL;DR
This paper introduces a novel typo correction system specifically designed for Turkish, a morphologically rich language, and integrates it into an e-commerce search engine. The system achieves state-of-the-art performance by employing a morphology-centered approach for candidate generation and a multi-source scoring function, outperforming existing methods and significantly improving search relevancy and conversion rates.
What Problem Does This Paper Address?
Typo correction is crucial for e-commerce search engines, as misspellings can lead to irrelevant search results and lost revenue. While existing typo correction methods are largely effective for languages like English, they struggle with morphologically rich languages (MRLs) such as Turkish. MRLs present unique challenges due to their complex word structures, agglutination, and diacritics, leading to high data sparsity and significant edit distances between misspelled and correct words. This paper tackles the problem of developing an effective typo corrector for Turkish within the demanding context of an e-commerce search engine, aiming to improve product relevancy without introducing significant latency.
Methods
The proposed system follows a two-stage approach: candidate generation and candidate ranking.
- Candidate Generation: This stage leverages a character-level transformer model trained with a contrastive learning objective. This model learns syntax-aware word representations, capturing intrinsic character patterns and language-specific rules (like lenition and vowel harmony). This approach allows for efficient retrieval of semantically similar words, including inflected forms, which is crucial for MRLs. The goal is to generate candidate corrections that are syntactically close to the misspelled word.
- Candidate Ranking: A multi-source scoring function is employed to rank the generated candidates. This function combines three key features:
- Edit Distance: Measures the similarity between the original query and the candidate query, particularly useful when multiple misspellings occur.
- Vector Similarity: Utilizes the syntax-aware word embeddings generated in the candidate generation phase, which are effective for phonetic and diacritic-based misspellings.
- Language Score (Perplexity): Two statistical language models (one trained on search logs, the other on product titles) are used to assess the likelihood of a candidate query, biasing towards correct inflections.
The weights for these scoring features are learned using a regression model.
Datasets
- Training Data: Automatically generated from search logs, incorporating user interactions (clicks) and synthetic data. Positive pairs are derived from user feedback chains (query refinement, fuzzy match and click) and synthetic generation (deacritization, insertion, deletion, replacement, swapping). Morphology-based pairs are generated using FastText embeddings on search logs.
- Evaluation Data:
- In-domain: A gold-standard test set of 5.6K queries, manually annotated by three native speakers with domain expertise and NLP background.
- Out-of-domain: The
trspell10dataset.
Key Insights and Ideas
- Morphology is Key for MRLs: The paper strongly emphasizes that simply applying English-centric typo correction methods is insufficient for MRLs like Turkish. Capturing morphological variations and inflections is paramount for accurate correction and improved search relevancy.
- Syntax-Aware Embeddings for Efficiency: Using character-level transformers to learn syntax-aware embeddings offers a balance between capturing linguistic nuances and maintaining low inference latency, making it suitable for real-time search engine applications.
- Multi-Source Scoring for Robustness: Combining edit distance, vector similarity, and language modeling provides a more robust ranking mechanism, addressing different types of errors and query complexities.
- Automated Data Generation is Feasible: The paper demonstrates that high-quality training data for typo correction can be effectively generated automatically from user logs and synthetic methods, reducing the reliance on expensive manual annotation.
Key Results
- Offline Performance: The proposed corrector achieved an F1-score of 86% on the in-domain evaluation, with 89% precision and 83% recall. On the
trspell10dataset, it achieved an F1-score of 98.18%, outperforming other listed correctors in terms of Spell Correction Accuracy (SCA). - Search Relevancy: The corrector improved the Normalized Discounted Cumulative Gain (NDCG) score from 90.9 to 91.2 on a set of 1000 queries. For misspelled samples, the NDCG score improved from 87.8 to 89.7.
- Online Performance: An A/B test showed a 6.99% increase in Conversion Rate (CR), with the new model achieving a CR of 4.84% compared to the old model's 4.52%.
Limitations and Future Work
- Word-Level Operation: The current corrector operates at the word level, which can lead to sub-optimal performance when context is crucial for correction.
- Foreign Word Adaptations: The system exhibits relatively weaker performance in correcting Turkish adaptations of foreign words that deviate from Turkish phonetic harmonies.
Future work includes:
- Modifying the candidate generation to a sequence-level model to better capture context.
- Adapting the approach to develop corrector models for other languages, such as Russian and Arabic.
Summary updated