TL;DR
This paper tackles the challenge of translating noisy, user-generated product reviews from English to Hindi. They introduce a novel approach that leverages character-level embeddings to map noisy words to their correct forms before translation, significantly improving accuracy on this difficult task.
What Problem Does This Paper Address?
The explosion of e-commerce has led to a massive increase in user-generated product reviews. While these reviews are invaluable for consumers, they are predominantly in English, creating a barrier for the growing number of non-English speakers, particularly in multilingual countries like India. Existing machine translation (MT) systems struggle with the inherent "noise" in these reviews, such as misspellings, ungrammatical constructions, and colloquialisms. This noise can lead to out-of-vocabulary (OOV) tokens, resulting in incorrect or missing translations, ultimately degrading the overall quality of the translated content. The paper aims to build a robust English-to-Hindi MT system specifically for product reviews that can effectively handle this noisy text.
Methods
The core of the paper's contribution lies in its novel approach to handling noisy tokens during inference. Instead of relying solely on standard word segmentation techniques like Byte-Pair Encoding (BPE), they introduce a two-stage process:
- Character-based Language Model and Word Vector Generation: They train a character-level language model (using an LSTM architecture) that learns to generate vector representations (embeddings) for words. This model is trained to map similar word pairs (a correct word and its noisy variant) closer together in a vector space, while pushing dissimilar pairs further apart. This is achieved by creating synthetic noisy versions of words and training the model to recognize them as related to their original forms.
- Noisy Token Replacement at Inference: At inference time, before feeding a sentence to the main NMT model, they check each token. If a token is not in the NMT model's vocabulary (an OOV token), they use the trained character-based model to find the most similar token from the source vocabulary in the vector space. This similar token then replaces the noisy one. This effectively "corrects" the noisy input before it reaches the translation model.
They also describe the creation of a new English-Hindi parallel corpus for product reviews, which involved crawling reviews, pre-processing, filtering, and human post-editing.
Datasets
- English-Hindi Review Domain Parallel Corpus: This is a newly created dataset specifically for product reviews. It was built by crawling English reviews from Flipkart, followed by pre-processing, filtering, and human post-editing by linguistic experts. This corpus was split into training (13,000 samples), development (599 samples), and test (2,539 samples) sets.
- Back-translated (BT) Corpus: Monolingual Hindi sentences were crawled and then back-translated into English using an in-house Hindi-to-English MT system. This synthetic corpus was augmented with the human-corrected parallel corpus.
- Other Datasets for Evaluation:
- English-German: Europarl corpus from WMT 2014.
- English-Czech: IWSLT17 dataset.
- English-French: IWSLT17 and MTNT18 datasets.
- English-Hindi (general domain): IIT Bombay English-Hindi parallel corpus.
Key Insights and Ideas
- The "Noise" Problem is Significant: User-generated text is inherently noisy, and standard MT models, which rely on fixed vocabularies, struggle immensely with this. Even small amounts of noise can drastically degrade translation quality.
- Character-level Information is Key: By looking at the character composition of words, the system can infer relationships between correct and misspelled words, which is crucial for handling OOV tokens.
- Inference-time Correction is Effective: Instead of trying to train the NMT model on every possible noisy variant (which is infeasible), correcting the noise before translation at inference time is a practical and effective strategy.
- Domain-Specific Corpora Matter: Building a parallel corpus tailored to product reviews is essential for achieving good performance on this specific task, as generic domain corpora may not capture the nuances of review language.
Key Results
- English-to-Hindi Review Translation: The proposed "Base+BT" model achieved a BLEU score of 37.79, a significant improvement over the baseline model (33.26 BLEU) trained only on the human-corrected corpus.
- Robustness to Noise: When tested on noisy versions of test sets (with 20-40% injected noise), the proposed method consistently outperformed baseline models (SR and BPE) and even a synthetic noise injection method, demonstrating its robustness. For example, on En-Hi, the proposed method achieved a BLEU score of 35.09 compared to 33.48 for synthetic noise.
- Cross-Lingual Performance: The approach also showed improvements on other language pairs:
- En-De: 28.91 BLEU (1.05 improvement over synthetic noise)
- En-Fr: 34.68 BLEU (1.63 improvement over synthetic noise)
- En-Cs: 14.52 BLEU (1.94 improvement over synthetic noise)
- Human Evaluation: Human evaluators rated the proposed model's outputs as having good fluency and adequacy, with high inter-annotator agreement scores.
Limitations and Future Work
- Small Character Count Noisy Tokens: The paper acknowledges a limitation where very short noisy tokens (e.g., "lok" vs. "look") with similar character combinations might still be challenging to correct perfectly, as the character-based similarity might lead to an incorrect replacement.
- Future Work: The authors suggest further research into improving the robustness of the system, potentially by exploring more advanced character-level models or different noise injection strategies. The availability of their created parallel corpus is also a contribution that can facilitate future research in this area.
Summary updated