Typos Correction Training against Misspellings from Text-to-Text Transformers

Guicai Xie, Ke Zhang, Lei Duan, Wei Zhang, Zeqian HuangCOLING 2024

TL;DR

This paper introduces ToCoTR, a novel end-to-end approach that integrates typos correction training directly into dense retrieval models. By training a T5-based dual-encoder to simultaneously correct misspellings and retrieve relevant documents, ToCoTR significantly improves retrieval effectiveness on typo-ridden queries, outperforming existing methods that rely on separate spell-checking stages or less direct typo-aware training.

What Problem Does This Paper Address?

Dense retrieval (DR) systems are crucial for information seeking, but their performance plummets when faced with queries containing typos. These errors, common in real-world search scenarios, create a mismatch between the query embedding and the intended meaning, leading to a significant drop in retrieval effectiveness. While previous work has explored various typo-aware training strategies, they often fall short of achieving the robustness of a two-stage approach (spell-checker + DR) or require complex training pipelines. This paper aims to bridge this gap by proposing a unified, end-to-end training framework that directly teaches DR models to handle misspellings.

Methods

The core of ToCoTR lies in its integration of typos correction training into a T5-based dual-encoder architecture. T5, being an encoder-decoder model, is well-suited for sequence-to-sequence tasks like spelling correction.

  • T5 Dual-Encoder Architecture: The authors leverage a pre-trained T5 model as the backbone for both query and passage encoders. The first token prediction from the decoder is used as the sentence representation for retrieval.
  • Typos Correction Training: This is formulated as a sequence-to-sequence task. The model is trained to predict the correct version of a text given a typo-ridden version. This is achieved using a teacher forcing negative log-likelihood loss.
  • Prompt-based Augmentation: To further enhance the alignment between typoed and correct words, a novel prompt-based augmentation technique is introduced. This technique adds special tokens around misspelled words, guiding the model to focus on correcting them.
  • Training Strategies: Two main training strategies are explored:
  • Joint Training: The retrieval loss (NLL) and typos correction loss are trained simultaneously.
  • Two-Stage Training: The model is first trained for typos correction, and then these learned weights are used to initialize the DR model for retrieval training. The paper finds the two-stage approach to be more effective.
  • Additional Techniques: ToCoTR also incorporates hard negative mining and self-teaching, which are known to improve DR performance.

Datasets

The experiments were conducted on three public datasets:

  • MS MARCO v1: A large-scale dataset for passage ranking, sampled from Bing search logs.
  • TREC Deep Learning Track Passage Retrieval Task 2019 (TREC 2019): Similar to MS MARCO but with different query annotations.
  • ANTIQUE: A dataset for non-factual question answering, featuring manually validated typoed queries.

For MS MARCO and TREC 2019, typoed queries were generated using five simulated typo generators. ANTIQUE provided pre-existing typo variations.

Key Insights and Ideas

  • End-to-End Typos Correction is Beneficial: Directly training a dense retriever to correct typos, rather than relying on separate pre-processing steps or indirect typo-aware methods, leads to significant improvements in robustness.
  • T5's Encoder-Decoder Architecture is Key: The inherent sequence-to-sequence capability of T5 makes it a suitable choice for integrating typos correction into the retrieval pipeline.
  • Prompt-based Augmentation Helps: The novel prompt-based technique, while not a massive gain on its own, demonstrably aids in aligning typoed words with their correct counterparts.
  • Two-Stage Training Outperforms Joint Training: Pre-training for typos correction and then fine-tuning for retrieval (two-stage) yields better results than jointly optimizing both objectives. This is attributed to the limitations of using only the first token's representation for correction in a joint setting.
  • Sophisticated Training Matters: Advanced training techniques like hard negative mining and self-teaching, when combined with typos correction, further boost performance, explaining why some complex retrievers outperform simpler typo-aware methods.

Key Results

  • Significant Improvement on Typoed Queries: ToCoTR dramatically outperforms other methods, including sophisticated typo-aware retrievers and combinations of spell-checkers with DR models, on queries with typos. For instance, on MS MARCO, ToCoTR achieves an MRR@10 of 31.3 with typos, compared to much lower scores for other typo-aware methods.
  • Competitive Performance on Clean Queries: ToCoTR also shows competitive or superior performance on queries without typos compared to other advanced retrievers.
  • Outperforms Spell-Checker Pipelines: ToCoTR significantly outperforms pipelines that use separate spell-checkers (like pyspellchecker, MS-Spellchecker, and GG-Spellchecker) followed by a DR model, demonstrating the advantage of an integrated approach.
  • Impact of Typos Proportion: The best performance for typos correction training is achieved with a typos proportion of 0.2 per input text.

Limitations and Future Work

  • Encoder-Only Architecture Limitation: ToCoTR, in its current form, cannot be directly applied to common encoder-only DR architectures due to its reliance on the encoder-decoder structure for typos correction.
  • Computational Cost: While efficient, the two-stage training process for ToCoTR incurs additional computational costs.
  • Future Work: The authors plan to:
  • Integrate typos correction training into multi-stage retrieval processes.
  • Explore training typos correction in encoder-only architectures.
  • Investigate the impact of typos correction training on other advanced retrieval models like coCondenser and SimLM.

Summary updated

BibTeX
@inproceedings{xie2024typos,
  author = {Guicai Xie and Ke Zhang and Lei Duan and Wei Zhang and Zeqian Huang},
  title = {{Typos Correction Training against Misspellings from Text-to-Text Transformers}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure