Abstract
A popular application of machine translation (MT) is gisting: MT is consumed as is to make sense of text in a foreign language. Evaluation of the usefulness of MT for gisting is surprisingly uncommon. The classical method uses reading comprehension questionnaires (RCQ), in which informants are asked to answer professionally-written questions in their language about a foreign text that has been machine-translated into their language. Recently, gap-filling (GF), a form of cloze testing, has been proposed as a cheaper alternative to RCQ. In GF, certain words are removed from reference translations and readers are asked to fill the gaps left using the machine-translated text as a hint. This paper reports, for the first time, a comparative evaluation, using both RCQ and GF, of translations from multiple MT systems for the same foreign texts, and a systematic study on the effect of variables such as gap density, gap-selection strategies, and document context in GF. The main findings of the study are: (a) both RCQ and GF clearly identify MT to be useful, (b) global RCQ and GF rankings for the MT systems are mostly in agreement, (c) GF scores vary very widely across informants, making comparisons among MT systems hard, and (d) unlike RCQ, which is framed around documents, GF evaluation can be framed at the sentence level. These findings support the use of GF as a cheaper alternative to RCQ.
Introduction
While much of machine translation (MT) research focuses on "dissemination"—the production of high-quality text for publication—the most prevalent real-world application is "gisting" (or assimilation). In gisting, users utilize MT to grasp the core meaning of a foreign-language text. Despite its importance, evaluating MT for gisting is often overlooked due to the high costs associated with traditional methods.
The established gold standard for measuring gisting quality is the Reading Comprehension Questionnaire (RCQ). This method involves human participants reading MT output and answering questions about the content. However, RCQs are resource-intensive, requiring professional question-writing, reference translations for control groups, and manual grading. This study investigates whether Gap Filling (GF)—a cloze-test variant where users fill in missing words in a reference translation using MT output as a "hint"—can serve as a cost-effective alternative.
Methodology
The researchers performed a head-to-head comparison between RCQs and GF tasks using four MT systems: Google Translate, Bing, Systran, and a "homebrew" Moses baseline. The study utilized an extended version of the CREG-mt-eval corpus, comprising 108 German documents with professional English translations.
Key innovations in the GF methodology included:
- Automated Gap Selection: Utilizing a statistical language model to identify "high-entropy" words that are difficult to predict solely from context.
- Variable Density: Testing gap densities of 10% and 20% to determine the impact of missing information on task difficulty.
- Contextual Hints: Comparing the effectiveness of providing a single machine-translated sentence versus the entire document as a hint.
Comparative Performance and Key Results
The study's primary finding is that RCQ and GF rankings generally align, both identifying commercial systems as superior to the Moses baseline. Quantitative results demonstrate that MT provides a significant boost to comprehension. Without an MT hint, users succeeded in gap-filling only 19-25% of the time. With an MT hint, success rates rose to approximately 58% for exact matches and up to 75% when allowing for synonyms.
Interestingly, a 20% gap density proved easier for participants than a 10% density. This "Density Paradox" occurs because the 20% version includes the difficult gaps from the 10% version plus additional, lower-entropy gaps that are easier to guess, thereby raising the overall average score. Furthermore, providing full document context did not significantly improve scores over single-sentence hints, suggesting that GF evaluation can be efficiently conducted at the sentence level.
Conclusion
The findings support the adoption of Gap Filling as a viable and much cheaper alternative to Reading Comprehension Questionnaires for evaluating MT gisting quality. GF tasks are faster to generate through automated "punching" of gaps and offer objective scoring. While high variance among human informants necessitates larger participant groups, the consistency of system rankings with traditional RCQs makes GF a promising tool for the MT research community. Future work may focus on aligning GF sentence selection more closely with the specific facts targeted by RCQ questions to further refine the correlation between these two evaluation methods.
Summary updated