A Comparative Evaluation of Data-driven Models in Translation Selection of Machine Translation

Yuseop Kim, Jeong Ho Chang, Byoung-Tak ZhangCOLING 2002

Introduction

In English-to-Korean machine translation, selecting the correct target word often requires deep contextual understanding. This paper evaluates the effectiveness of latent semantic spaces—specifically Latent Semantic Analysis (LSA) and Probabilistic Latent Semantic Analysis (PLSA)—in resolving word-sense ambiguity. By leveraging data-driven models, the researchers aim to overcome the "data sparseness" problem inherent in traditional dictionary-based systems.

The Challenge of Data Sparseness

Traditional machine translation systems rely on dictionaries of co-occurrence pairs. However, these systems frequently encounter data sparseness, where a specific word combination (e.g., a verb and its object) is missing from the training data. For instance, the verb "build" has multiple Korean translations depending on its object, such as a plant, a car, or a company.

Examples of the verb 'build' translating to different Korean senses based on context.

When a system encounters an unseen pair like "build + vehicle," it cannot rely on exact dictionary matches. Manually constructing comprehensive thesauri to cover every possible combination is prohibitively expensive. This research explores how models can "learn" semantic similarities from raw text, allowing the system to infer the correct translation by looking at semantically similar neighbors in a latent space.

Methodology: LSA vs. PLSA

The authors implemented a three-step translation selection process:

  1. Dictionary Lookup: The system first attempts an exact match for the word pair (e.g., verb-object) in a pre-built dictionary.
  2. Latent Space Mapping: If no match is found, the system turns to data-driven models to find semantically related words.
  3. LSA (Latent Semantic Analysis): Uses Singular Value Decomposition (SVD) to reduce a word-document matrix into a lower-dimensional space where words with similar contexts are placed close together.
  4. PLSA (Probabilistic LSA): A statistical approach that uses an "aspect model" and the Expectation-Maximization (EM) algorithm to define a generative model of the data.
  5. k-Nearest Neighbor (k-NN) Learning: The system identifies the $k$ most similar words to the unknown argument in the latent space and selects the translation associated with the majority of those neighbors.

Comparative Performance Analysis

The models were trained on the 1988 AP News corpus and evaluated using grammatical relations extracted from the Wall Street Journal corpus. Both LSA and PLSA significantly outperformed the baseline dictionary-only approach, providing accuracy boosts of 11% to 20%.

Accuracy comparison for verb-object relations across baseline, LSA, and PLSA models.

As shown in the results for verb-object relations, PLSA consistently outperformed LSA. While LSA's performance was sometimes inconsistent as the number of neighbors ($k$) increased, PLSA demonstrated a more stable improvement, reaching 86.05% accuracy at $k=10$. This suggests that PLSA's probabilistic foundation creates a more coherent semantic neighborhood than LSA's linear algebraic approach. Similar trends were observed in subject-verb and adjective-noun relations, where PLSA reached peak accuracies of 87.49% and 82.76%, respectively.

Conclusion and Limitations

While PLSA is superior in accuracy, it comes with a significant computational trade-off, taking approximately 50 times longer to train than LSA due to the iterative nature of the EM algorithm. Other challenges identified include aggressive stemming errors and vocabulary constraints. Future research will focus on more robust morphological analysis and automated methods for determining the optimal value for $k$ based on dataset size.

Summary updated

BibTeX
@inproceedings{kim2002comparative,
  author = {Yuseop Kim and Jeong Ho Chang and Byoung-Tak Zhang},
  title = {{A Comparative Evaluation of Data-driven Models in Translation Selection of Machine Translation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2002},
}
Figure