Tuning SMT with a Large Number of Features via Online Feature Grouping

Lemao Liu, Tiejun Zhao, Taro Watanabe, Eiichiro SumitaIJCNLP 2013 · OpenReview

Abstract

Lemao Liu, Tiejun Zhao, Taro Watanabe, Eiichiro Sumita. Proceedings of the Sixth International Joint Conference on Natural Language Processing. 2013.

This paper introduces a novel tuning method for Statistical Machine Translation (SMT) that handles hundreds of thousands of features without succumbing to overfitting. By using the OSCAR algorithm to automatically group sparse features during an online learning process, the researchers achieved significant gains in translation quality (BLEU scores) while drastically reducing model complexity.

Introduction

In Statistical Machine Translation (SMT), "tuning" involves adjusting the weights of various features—such as word probabilities or sentence lengths—to optimize translation quality. Modern SMT models increasingly rely on "fine-grained" features, such as specific translation rules or n-grams, which can scale to hundreds of thousands of parameters.

However, this high dimensionality leads to a "sparsity pitfall":

  1. Overfitting: Most fine-grained features appear in the large training corpus but are absent from the small development (tuning) set. If a feature does appear in the tuning set, the model may over-rely on it, resulting in poor generalization to unseen test data.
  2. The Tuning Dilemma: Tuning on the entire training set to capture more features is often impractical. Tuning requires high-quality "multi-reference" translations and domain-specific data, which are typically unavailable for the general training corpus.

Methodology

The authors propose a two-step tuning framework centered on automated Feature Grouping.

OSCAR and Online Learning

The core of the approach is the Octagonal Shrinkage and Clustering Algorithm for Regression (OSCAR). Unlike standard $L_1$ regularization (Lasso), which primarily drives weights to zero, OSCAR encourages the weights of similar features to be exactly equal, effectively clustering them into groups.

To scale this to approximately 500,000 features, the authors integrated OSCAR into the Forward-Backward Splitting (FOBOS) framework. This allows for online weight updates, processing data incrementally rather than in a single batch, which ensures computational feasibility. The authors also proved a mathematical lemma showing that if the input features are sparse, the optimal solution remains sparse, allowing the complexity to depend on active features rather than the total feature count.

Algorithm 1: The OSCAR optimization procedure for feature grouping.

The Two-Step Workflow

  1. Grouping: The model learns the group structure of atomic features (e.g., specific rule IDs) using the large training dataset.
  2. Tuning: Each identified group is treated as a single feature. The model is then tuned on a high-quality development set using standard tools like Pairwise Ranking Optimization (PRO).

Experimental Setup

The method was evaluated on the IWSLT 2008 Chinese-to-English translation task:

  • Data: 30,000 bilingual sentence pairs for training.
  • Evaluation: IWSLT 2003, 2004, and 2008 sets, featuring 16 reference translations per sentence for robust evaluation.
  • Feature Scale: The model utilized roughly 500,000 rule ID features.

Results

The proposed OSCAR method outperformed state-of-the-art tuning methods, including MERT and standard PRO, across several metrics.

Comparison of OSCAR against baseline tuning methods on the IWSLT 2008 task.

Translation Quality and Complexity

As shown in the results table, OSCAR achieved a 41.8 BLEU score on the test set, surpassing MERT (40.6), standard PRO (41.1), and $L_1$ regularization (41.0). Notably, OSCAR achieved this with a much "leaner" model, using only 503 active grouped features, compared to the 11,081 active features required by standard PRO with rule IDs.

Generalization

A key finding was the "reuse" rate of features. Approximately 84.5% (425 out of 503) of the features OSCAR activated during tuning were also utilized during testing. In contrast, other methods showed significantly lower reuse rates, suggesting they were overfitting to the tuning data. By grouping features, the model leverages the "collective signal" of rare features; even if a specific rule is missing from the tuning set, its group-mates likely provide enough information to assign a meaningful weight.

Discussion and Conclusion

While OSCAR provides superior translation quality and model sparsity, it comes with a higher computational cost. The total runtime for OSCAR was 1,256 minutes, compared to just 34 minutes for standard PRO, primarily due to the multiple decoding passes required over the training data.

Future work will focus on distributed learning to improve scalability for industrial-scale datasets. Additionally, the researchers plan to investigate the behavior of feature grouping across different language pairs and translation architectures beyond hierarchical phrase-based models.

Summary updated

BibTeX
@inproceedings{liu2013tuning,
  author = {Lemao Liu and Tiejun Zhao and Taro Watanabe and Eiichiro Sumita},
  title = {{Tuning SMT with a Large Number of Features via Online Feature Grouping}},
  booktitle = {IJCNLP},
  year = {2013},
  url = {https://openreview.net/forum?id=KDcBoDknRZ6},
}
Figure