TL;DR
This paper investigates "gap-filling" (GF) as a more cost-effective alternative to traditional "reading comprehension questionnaires" (RCQ) for evaluating machine translation (MT) used for "gisting" (understanding the gist of a foreign text). The study finds that GF is a viable and cheaper method, largely agreeing with RCQ in ranking MT systems, while offering more flexibility in evaluation scope.
What Problem Does This Paper Address?
Machine translation (MT) is widely used for "gisting," where users consume raw MT output to understand foreign language texts without post-editing. However, evaluating the usefulness of MT for this specific application is surprisingly uncommon and often relies on expensive methods. The traditional approach, Reading Comprehension Questionnaires (RCQ), involves asking informants to answer questions about a machine-translated text. This requires professionally written questions and human translations for control groups, making it costly and time-consuming. The paper explores "gap-filling" (GF), a type of cloze test, as a cheaper and more efficient alternative for evaluating MT's gisting capabilities.
Methods
The core of this research is a comparative evaluation of two methods: Reading Comprehension Questionnaires (RCQ) and Gap-Filling (GF).
- RCQ: Informants were given machine-translated or professionally translated documents and asked to answer questions about them. The performance on these questions was used to assess the quality of the MT. The paper details different scoring methods for RCQ, including simple averaging, weighted averaging (giving more importance to inference and reorganization questions), and literal-only scoring.
- GF: In this method, words are removed from reference translations, creating "gaps." Informants are then asked to fill these gaps using the machine-translated text as a hint. The success rate (proportion of correctly filled gaps) serves as the evaluation metric. The paper systematically studies various GF configurations, including:
- Hint availability: With or without the MT output as a hint.
- Gap selection strategies: Random placement versus placement based on language model entropy (aiming for harder-to-guess words).
- Gap density: The percentage of words removed (10% vs. 20%).
- Context: Using a single sentence as a hint versus the entire machine-translated document.
Datasets
The study utilizes an extended version of the CREG-mt-eval corpus. This corpus is derived from the expert-built CREG reading comprehension corpus, originally designed for learners of German.
- It contains 108 German documents across various domains (literature, news, job adverts), averaging 372 words and 33 sentences per document.
- These documents were machine-translated into English using four systems: a baseline statistical phrase-based Moses, Google Translate, Bing, and Systran.
- Professional human translations for a subset of 36 documents were also available for control purposes.
- The original German questions from the CREG corpus were professionally translated into English for the RCQ task.
Key Insights and Ideas
- GF is a viable, cheaper alternative: The study strongly supports GF as a more cost-effective method for evaluating MT for gisting compared to RCQ, requiring less manual preparation and potentially enabling crowdsourcing.
- MT is useful for gisting: Both RCQ and GF clearly demonstrate that MT output is useful for understanding foreign language texts.
- Ranking agreement: Global rankings of MT systems by RCQ and GF scores largely agree, suggesting GF captures similar quality judgments.
- Informant variability in GF: GF scores show significant variation across individual informants, making direct comparisons between MT systems challenging based on average scores alone. However, individual informants tend to be consistent in their relative ranking of systems.
- Sentence-level evaluation flexibility: Unlike RCQ, which is typically document-focused, GF evaluation can be effectively framed at the sentence level, simplifying the task and potentially improving efficiency.
- Entropy-based gap selection is better: Using language model entropy to select gaps makes the GF task harder when no MT hint is provided, but this effect diminishes when MT is used as a hint.
- Higher gap density can be easier (with MT): Counter-intuitively, a higher gap density (20%) led to higher success rates when MT was used as a hint. This is explained by the inclusion of easier gaps alongside harder ones.
- Context doesn't significantly help (or hurt): Providing the entire document as a hint for GF did not significantly improve performance over providing just the problem sentence.
Key Results
- MT usefulness: RCQ scores for machine-translated texts reached 70-95% of scores for professional translations. GF success rates improved by about 3 times when MT was used as a hint compared to no hint.
- System ranking: Both RCQ and GF consistently ranked the homebrew Moses system as the worst. Commercial systems like Google and Bing generally performed best.
- GF success rates: With MT hints, average GF success rates were around 58%, significantly higher than the 19% achieved without hints using entropy-based gap selection.
- Inter-annotator agreement: Pairwise Pearson correlations for GF success averaged around 0.58, indicating good inter-annotator agreement.
- Effect of gap density: With MT hints, 20% gap density yielded higher success rates than 10% density.
- Synonym credit: Allowing credit for synonyms increased GF success rates considerably (e.g., from 58.9% to 75.5% for all systems averaged), but the relative ranking of MT systems remained largely the same.
Limitations and Future Work
- Informant variability: The wide variation in GF scores across informants makes it difficult to draw definitive conclusions about MT system differences solely from average scores.
- Correlation between GF and RCQ: While rankings generally agree, the correlation between specific GF and RCQ scores for document-MT system pairs was found to be low. This is attributed to the scarcity of data points per pair and potential mismatches between the sentences selected for GF and the content targeted by RCQ questions.
- Future Work: The authors suggest using sentences for GF that are directly related to the RCQ tests to improve correlation. They also note that the discriminative power of both RCQ and GF evaluations might increase with larger datasets, making GF a more feasible option for collecting such data. Further analysis of actual GF results and exploring synonym handling more deeply are also suggested.
Summary updated