A Comparative Study on Language Model Adaptation Techniques Using New Evaluation Metrics

Hisami Suzuki, Jianfeng GaoEMNLP 2005 · OpenReview

Abstract: This paper presents comparative experimental results on four techniques of language model adaptation, including a maximum a posteriori (MAP) method and three discriminative training methods—the boosting algorithm, the average perceptron, and the minimum sample risk method—on the task of Japanese Kana-Kanji conversion. We evaluate these techniques beyond simply using the character error rate (CER): the CER results are interpreted using a metric of domain similarity between background and adaptation domains and are further evaluated by correlating them with a novel metric for measuring the side effects of adapted models. Using these metrics, we show that the discriminative methods are superior to a MAP-based method not only in terms of achieving larger CER reduction but also in being more robust against the similarity of background and adaptation domains, achieving larger CER reduction with fewer side effects.

Introduction

Language models (LMs) are essential for Input Method Editors (IMEs), which convert phonetic Japanese strings (Kana) into Kanji characters. However, LMs are notoriously brittle when applied to domains outside their training data. Cross-domain adaptation addresses this by fine-tuning a large "background" model with a smaller "target" domain dataset. Traditionally, these techniques are evaluated using Character Error Rate (CER). However, CER alone does not account for "side effects"—new errors introduced in previously correct conversions—which can significantly degrade user experience. This study evaluates four adaptation techniques using novel metrics for domain similarity and backward compatibility.

Methodology

We compare a generative baseline against three discriminative approaches:

  • Linear Interpolation (LI): A standard MAP-based approach that merges probabilities from background and adaptation models.
  • Discriminative Methods: These algorithms directly minimize conversion errors on adaptation data. We evaluate Boosting (exponential loss), Average Perceptron (minimum square error loss), and Minimum Sample Risk (MSR) (direct error count minimization via line-search).

To measure side effects, we introduce the Error Ratio (ER): $$ER = \frac{\text{New Errors Introduced}}{\text{Old Errors Corrected}}$$ An ER below 1.0 signifies that the model corrects more errors than it creates, which is vital for maintaining user trust in software updates.

The Error Ratio (ER) vs. Relative Error Reduction (RER) plot illustrates the trade-off between accuracy gains and side effects. The shaded regions represent undesirable performance (ER > 1 or RER < 0).

Experimental Setup

The background model was trained on the Nikkei newspaper corpus (36M words). Adaptation was tested across four domains with varying similarity to the background: Yomiuri (newspaper), TuneUp (balanced), Encarta (encyclopedia), and Shincho (novels). For each domain, 8,000 sentences were used for training, with separate sets for testing and tuning.

Results and Discussion

Discriminative methods significantly outperformed linear interpolation across all domains. A key finding was the robustness of discriminative methods to domain similarity. For the Yomiuri domain, which is highly similar to the background Nikkei data, linear interpolation yielded a 0% CER reduction. In contrast, discriminative methods achieved approximately 22% relative error reduction (RER) on the same data.

Performance comparison across four adaptation domains (Yomiuri, TuneUp, Encarta, and Shincho) shows discriminative methods consistently outperforming linear interpolation.

The ER metric revealed that discriminative methods are far more effective at preserving backward compatibility. When plotting RER against ER, methods like MSR consistently clustered in the high-reduction, low-side-effect region (bottom-right), whereas LI often resulted in higher ERs, sometimes exceeding 1.0. This suggests that discriminative methods focus on correcting specific errors rather than just shifting general probability distributions, making them more suitable for user-facing software.

Conclusion

Discriminative adaptation techniques, particularly MSR and Perceptron, offer a superior balance of accuracy and stability compared to traditional generative methods. By minimizing side effects, these methods ensure a more consistent user experience during model updates. Future work will explore applying these techniques to incremental learning scenarios, where minimizing side effects is critical to maintaining user trust during real-time adaptation.

Summary updated

Abstract

This paper presents comparative experimental results on four techniques of language model adaptation, including a maximum a posteriori (MAP) method and three discriminative training methods, the boosting algorithm, the average perceptron and the minimum sample risk method, on the task of Japanese Kana-Kanji conversion. We evaluate these techniques beyond simply using the character error rate (CER): the CER results are interpreted using a metric of domain similarity between background and adaptation domains, and are further evaluated by correlating them with a novel metric for measuring the side effects of adapted models. Using these metrics, we show that the discriminative methods are superior to a MAP-based method not only in terms of achieving larger CER reduction, but also of being more robust against the similarity of background and adaptation domains, and achieve larger CER reduction with fewer side effects.
BibTeX
@inproceedings{suzuki2005comparative,
  author = {Hisami Suzuki and Jianfeng Gao},
  title = {{A Comparative Study on Language Model Adaptation Techniques Using New Evaluation Metrics}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2005},
  url = {https://openreview.net/forum?id=H1NU_mWu-r},
}

Loading related content...

Figure