Stream-based Translation Models for Statistical Machine Translation

Abby Levenberg, Chris Callison-Burch, Miles OsborneNAACL 2010

TL;DR

This paper introduces a novel approach to statistical machine translation (SMT) that can efficiently adapt to continuously arriving parallel text data, a common scenario in real-world applications. By using an online EM algorithm and a dynamic suffix array, they achieve performance comparable to traditional batch-trained models but with significantly reduced computational overhead and memory usage, making SMT more practical for dynamic data streams.

What Problem Does This Paper Address?

Traditional statistical machine translation (SMT) systems rely on large, static datasets of parallel text (e.g., translated documents) for training. However, many real-world scenarios involve a continuous, unbounded stream of new parallel data, such as daily governmental proceedings, ongoing translation agency output, or crowd-sourced translations. Retraining SMT models from scratch with this ever-growing data is computationally prohibitive, requiring days of processing. Furthermore, storing the entire accumulated data for retraining becomes a significant memory challenge. This paper addresses the need for SMT systems that can efficiently and effectively incorporate this incoming stream of data without the massive computational and memory costs associated with traditional batch retraining.

Methods

The core of the proposed solution lies in two key innovations:

  • Stepwise Online EM (sOEM) for Incremental Retraining: Instead of retraining the entire model on the whole dataset, sOEM incrementally updates the translation model parameters. It processes data in "mini-batches" (small sets of new sentences) and uses an online EM algorithm. This algorithm retains expected counts between iterations and interpolates new information with existing statistics, allowing the model to adapt to new data without re-processing everything. This is a departure from traditional batch EM, which requires all data to be available at each iteration.
  • Dynamic Suffix Arrays for Bounded Space: To manage the memory requirements of potentially unbounded data streams, the authors employ dynamic suffix arrays. Suffix arrays are efficient data structures for searching text. A dynamic variant allows for both insertions and deletions. This enables the system to maintain a fixed-size representation of the parallel sentences by adding new ones and deleting an equivalent number, thus bounding the memory footprint while still incorporating recent data.

Datasets

The experiments were conducted using the Europarl corpus, specifically the French-English and German-English language pairs. The data was treated as a continuous stream, with different "epochs" representing distinct time periods within the corpus. For evaluation, held-out test documents were used. The authors did not introduce new benchmarks but rather adapted existing datasets to simulate a streaming scenario.

Key Insights and Ideas

  • Recency Matters: The paper strongly supports the intuition that using more recent training data significantly improves translation performance. This is demonstrated by showing that models retrained on chronologically closer data outperform static baselines.
  • Incremental Adaptation is Key: The computational burden of retraining SMT models on massive, growing datasets is a major bottleneck. The sOEM approach offers a practical solution by enabling efficient, incremental updates.
  • Space Efficiency is Crucial: For true stream processing, managing memory is as important as computational efficiency. Dynamic suffix arrays provide a mechanism to bound memory usage while still benefiting from new data.
  • Online EM Outperforms Batch (in practice): While both unbounded and bounded online EM models outperform static baselines, the bounded version, despite using less data per epoch, achieves comparable performance to batch models, highlighting the effectiveness of incremental updates and recency.

Key Results

  • Both unbounded and bounded online EM models outperform the static baseline in terms of BLEU scores.
  • The online models achieve comparable performance to batch retrained models but with significantly less computational overhead and memory usage.
  • The bounded online model (using a fixed window of data) achieves a +1.24 BLEU improvement over the static baseline for the final test point.
  • When combined with an adaptive language model (LM), the total gain over a static baseline can reach +2.32 BLEU.
  • The authors found no significant difference between various interpolation parameters tested for the sOEM algorithm, suggesting robustness.

Limitations and Future Work

  • Interpolation Parameter Sensitivity: While the authors found no significant difference in their experiments, they acknowledge that the choice of interpolation parameters could be explored further.
  • Future Work on Adaptation Strategies: The authors suggest investigating more sophisticated adaptation strategies that leverage feedback from the decoder to reinforce useful word alignments and delete bad ones.
  • Application to Real-World Scenarios: They plan to test the online EM setup in applications like computer-aided translation or with crowdsourced data from platforms like Amazon's Mechanical Turk.
  • Online Language Models: While the focus is on translation models, they note that online language models can also be used to adapt to new vocabulary from the input stream.

Summary updated

BibTeX
@inproceedings{levenberg2010stream,
  author = {Abby Levenberg and Chris Callison-Burch and Miles Osborne},
  title = {{Stream-based Translation Models for Statistical Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
}
Figure