YiSi - a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available Resources

Chi-kiu LoACL 2019

TL;DR

This paper introduces YiSi, a novel unified metric for evaluating machine translation (MT) quality that goes beyond simple word overlap to capture semantic meaning. YiSi is designed to be adaptable to languages with varying levels of linguistic resources, offering a more robust and portable alternative to existing metrics like BLEU.

What Problem Does This Paper Address?

Evaluating the quality of machine translation is a persistent challenge. Traditional metrics like BLEU, while widely used due to their simplicity and language independence, often fail to correlate well with human judgment of translation quality. This is because they primarily focus on surface-level word overlap and miss deeper semantic meaning. More advanced semantic metrics exist, but they often require extensive linguistic resources (like parsers or bilingual dictionaries) that are not available for all languages, limiting their applicability. The authors aim to bridge this gap by creating a unified metric that can leverage semantic information when available but gracefully fall back to simpler lexical similarity measures when resources are scarce.

Methods

YiSi is a framework that can be instantiated into different versions (YiSi-0, YiSi-1, YiSi-2) depending on the available resources. The core idea is to represent sentences using semantic frames and role fillers, and then align these structures between the machine translation (MT) output and the human reference.

  • Semantic Parsing: The first step involves applying a shallow semantic parser to both the MT output and the reference sentence.
  • Frame and Argument Alignment: A maximum weighted bipartite matching algorithm is used to align semantic frames based on predicate similarity and then to align arguments (role fillers) within those frames based on lexical similarity.
  • Weighted F-score: The overall quality is computed as a weighted f-score, combining precision and recall derived from the aligned semantic structures.
  • Lexical Similarity: The definition of lexical similarity varies:
  • YiSi-0: Uses longest common character substring for languages with no specific resources.
  • YiSi-1: Employs contextual embeddings from BERT (specifically, the 18th layer for English, 9th for Chinese, and 9th for multilingual) to capture word meaning in context. Lexical weights are based on inverse document frequency (IDF).
  • YiSi-2: A cross-lingual variant that uses bilingual embeddings to evaluate semantic similarity between source and target languages, also leveraging BERT embeddings.
  • Structural Semantic Similarity: YiSi can optionally incorporate structural semantic similarity by merging semantic role labels into broader categories (who, did, what, etc.) for more robust performance.
  • Parameterization: The final score is a weighted combination of precision and recall, with weights ($\alpha$ and $\beta$) that can be tuned for different use cases (e.g., MT evaluation vs. system optimization).

Datasets

The paper uses the WMT 2018 metrics task evaluation set for development experiments. The official human judgments from WMT 2018, collected via direct assessment, are used for evaluation. No new datasets are introduced.

Key Insights and Ideas

  • Contextual Embeddings are Key: The significant improvement in YiSi's correlation with human judgment when using BERT's contextual embeddings (YiSi-1) highlights the importance of capturing word meaning in context, a limitation of static word embeddings.
  • Unified Framework for Resource Diversity: The modular design of YiSi, allowing it to adapt to varying resource availability (from character-based similarity to deep contextual embeddings), is a crucial contribution for broader adoption.
  • Semantic Parsing Enhances Evaluation: The paper demonstrates that incorporating semantic frames and role alignment, even with shallow parsers, leads to better correlation with human judgment, especially for evaluating translation fluency.
  • YiSi-1_srl Outperforms: Adding structural semantic information (YiSi-1_srl) further boosts the correlation with human judgment, suggesting that understanding sentence structure is vital for accurate MT evaluation.

Key Results

  • System-Level Correlation: YiSi-0 showed stable performance across various translation directions. YiSi-1 significantly outperformed other metrics, including chrF and BLEU, in correlating with human judgment at the system level for many language pairs. YiSi-1_srl further improved these results.
  • Segment-Level Correlation: YiSi-0 performed comparably to chrF and BLEU. YiSi-1 again showed strong performance, outperforming most WMT2018 participants at the segment level for most translation directions. YiSi-1_srl also provided improvements.
  • Quality Estimation (YiSi-2): YiSi-2, the cross-lingual variant, showed weaker performance than YiSi-1, particularly when a reference translation in the same language was unavailable for fluency evaluation. However, YiSi-2_srl showed significant improvement in evaluating English-German translation without a reference, indicating the value of semantic parsing for fluency assessment.

Limitations and Future Work

  • Resource Dependency: While YiSi is designed to be adaptable, its performance is still influenced by the availability and quality of linguistic resources, particularly for semantic parsing.
  • YiSi-2 Performance: The paper notes that YiSi-2's performance is weaker than YiSi-1, especially in scenarios lacking a same-language reference for fluency. Future work could focus on improving cross-lingual semantic evaluation.
  • Parameter Tuning: The optimal values for parameters like $\alpha$ and $\beta$ might vary across different language pairs and MT systems, requiring further investigation.
  • Computational Cost: Using BERT embeddings and semantic parsers can be computationally more expensive than simpler metrics like BLEU, which might be a consideration for real-time applications.

Summary updated

BibTeX
@inproceedings{lo2019yisi,
  author = {Chi-kiu Lo},
  title = {{YiSi - a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available Resources}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure