Task-based Evaluation of Multiword Expressions: a Pilot Study in Statistical Machine Translation

Marine Carpuat, Mona T. DiabNAACL 2010 · OpenReview

Abstract

We conduct a pilot study for task-oriented evaluation of Multiword Expression (MWE) in Statistical Machine Translation (SMT). We propose two different integration strategies for MWE in SMT, which take advantage of different degrees of MWE semantic compositionality and yield complementary improvements in SMT quality on a large-scale translation task.

Introduction

In Statistical Machine Translation (SMT), systems typically treat language as a collection of phrases—sequences of words defined by statistical frequency in a bilingual corpus rather than linguistic meaning. This approach often fails to account for Multiword Expressions (MWEs), such as "kick the bucket" or "make a decision," which act as single semantic units. MWEs range from purely idiomatic to compositional, and standard SMT systems often break these units apart, resulting in unnatural translations. This study explores whether a broad, monolingual dictionary of MWEs can improve general-purpose SMT quality.

Methodology

The researchers integrated MWE knowledge from WordNet 3.0 into an English-to-Arabic SMT pipeline using two distinct strategies:

  1. Static Integration: MWEs are treated as "frozen" units. Before training, MWEs are identified and joined with underscores (e.g., written_report), forcing the system to treat them as single, indivisible tokens.
  2. Dynamic Integration: This method adds a feature to the translation lexicon that counts MWEs within a phrase. During decoding, the system uses this feature as a bias to prefer phrases that keep MWEs intact without strictly forbidding the system from breaking them if statistical evidence is strong.

To identify MWEs, the study employed a maximum forward match algorithm on WordNet 3.0, augmented with rules to handle morphological variations (e.g., "keep one's eyes peeled" vs. "kept her eyes peeled").

Experimental Results

The system was trained on a large-scale corpus of 2.5 million English-Arabic sentence pairs and evaluated using 813 newswire sentences from the 2008 NIST Open MT Evaluation. Performance was measured using BLEU (higher is better) and Translation Edit Rate (TER; lower is better).

Performance comparison of MWE integration strategies across different metrics.

As shown in the results table, both MWE integration strategies outperformed the baseline. Notably, WordNet-based MWEs yielded better scores than a baseline using the "top 500 most frequent n-grams," suggesting that linguistic cohesion is more valuable for translation than raw statistical frequency. The static approach provided the most significant boost to BLEU (31.27), while the dynamic approach was slightly more effective at reducing the TER (58.89).

Analysis of Integration Strategies

The study highlights that the optimal integration strategy depends on the MWE's position on the compositionality spectrum. Static integration is ideal for fixed, non-compositional expressions, while dynamic integration benefits compositional expressions by allowing the system to leverage word-for-word knowledge when appropriate.

Complementarity of static and dynamic integration strategies based on sentence-level impact.

An analysis of the two strategies revealed they are highly complementary. As illustrated in the table above, the strategies had opposite impacts on approximately 27% of the test sentences (16% where static helped but dynamic hurt, and 11% where dynamic helped but static hurt). This suggests that a hybrid approach leveraging both strategies could yield even greater improvements.

Conclusion and Future Work

This pilot study demonstrates that explicitly identifying MWEs significantly improves SMT quality, even in large-scale systems. Future research could improve MWE identification beyond simple dictionary lookups to reduce false positives and explore the inclusion of bilingual MWE knowledge or target-side (Arabic) MWEs to further enhance translation accuracy.

Summary updated

Abstract

We conduct a pilot study for task-oriented evaluation of Multiword Expression (MWE) in Statistical Machine Translation (SMT). We propose two different integration strategies for MWE in SMT, which take advantage of different degrees of MWE semantic compositionality and yield complementary improvements in SMT quality on a large-scale translation task.
BibTeX
@inproceedings{carpuat2010task,
  author = {Marine Carpuat and Mona T. Diab},
  title = {{Task-based Evaluation of Multiword Expressions: a Pilot Study in Statistical Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
  url = {https://openreview.net/forum?id=SyV4UQZuWH},
}
Figure