ILR-Based MT Comprehension Test with Multi-Level Questions

Douglas Jones, Martha Herzog, Hussny Ibrahim, Arvind Jairam, Wade Shen, Edward Gibson, Michael EmontsNAACL 2007

TL;DR

This paper introduces a novel comprehension test for machine translation (MT) that aligns with the Interagency Language Roundtable (ILR) proficiency scale. By evaluating how well users understand MT output across different difficulty levels and genres, they found that comprehension accuracy doesn't perfectly correlate with traditional translation error metrics, suggesting MT quality is a multi-faceted concept.

What Problem Does This Paper Address?

Evaluating machine translation (MT) quality has traditionally relied on metrics like BLEU, which measure n-gram overlap with reference translations, or fluency and adequacy scores. While useful, these don't always capture whether a human can actually understand the translated content. The Defense Language Proficiency Test (DLPT) is the gold standard for assessing human language learners' proficiency, and this paper adapts its principles to create a more robust MT evaluation that focuses on comprehension. Previous attempts, like the DLPTstar, showed promise but had limitations in matching MT capabilities to material difficulty. This work aims to refine that by creating a test specifically designed to probe MT performance across a range of ILR-defined difficulty levels and genres, addressing the question of whether current MT systems can truly convey meaning effectively.

Methods

The researchers developed a new comprehension test based on the DARPA GALE 2006 evaluation data, selecting about half of the available material. They focused on four genres: newswire, newsgroups, broadcast news, and talk radio, ensuring a substantial word count for each. The core innovation lies in constructing questions that map to multiple ILR difficulty levels (0+, 1, 1+, 2, 2+, 3) within each document. They then created two sets of questions for each document to allow for a balanced testing design.

Crucially, the test was administered to human subjects who read either a professional "Gold Standard" (GS) translation or an MT output (a merged output from three different MT systems). Subjects were given ample time (up to 5 hours) to complete the test. The scoring was done by two testing experts, with a high inter-rater agreement of 96%. They also compared comprehension scores against Human Translation Error Rate (HTER), a metric that counts human edits needed to correct MT output.

Datasets

The primary dataset used for this study was derived from the DARPA GALE 2006 evaluation data. Approximately half of this material was selected, comprising twenty-four test documents balanced across four genres: newswire, newsgroups, broadcast news, and talk radio. The total word count for the test material was approximately 12,200 words. No new datasets were introduced; rather, existing data was repurposed and structured for this specific evaluation.

Key Insights and Ideas

  • Comprehension is More Than Just Error Rate: The most striking finding is that comprehension accuracy does not strongly correlate with traditional translation error metrics like HTER. This suggests that MT quality is a nuanced concept, and systems can sometimes be understandable even with significant errors, or conversely, have low comprehension despite relatively low error rates.
  • "Robust" Translations Matter: The concept of "robust" translations emerged – where MT output has a high error rate (HTER) but still achieves good comprehension. This highlights the potential for MT systems to be useful in practice even if they aren't perfect, as human readers can leverage world knowledge to fill in gaps.
  • Genre and Modality Impact Comprehension: Text-based genres (newswire, newsgroups) generally yielded higher comprehension than audio-based genres (broadcast news, talk radio). Within text, more structured genres like newswire performed better than less structured ones like newsgroups.
  • Level 1 Comprehension is Challenging for MT: Even with materials designed to be at or near Level 1 difficulty, MT systems struggled more than expected, indicating a potential deficit in handling simpler, factual content.
  • MT Takes Longer to Read: Subjects consistently took longer to read MT output compared to Gold Standard translations, indicating an overhead associated with processing less fluent or less accurate text.

Key Results

  • Overall Comprehension: Subjects achieved 95% comprehension for Gold Standard (GS) translations and 74% for Machine Translation (MT).
  • Genre Performance: Comprehension accuracy varied by genre: Newswire (97% GS, 94% MT), Newsgroups (93% GS, 77% MT), Broadcast News (94% GS, 72% MT), Talk Radio (86% GS, 66% MT).
  • Level Performance: Comprehension accuracy by ILR level showed: L1~ (97% GS, 76% MT), L2 (96% GS, 82% MT), L2+ (91% GS, 77% MT), L3 (88% GS, 51% MT).
  • Time Overhead: It took approximately 15% more time to read MT output compared to GS translations.
  • Comprehension vs. Error Rate: A weak inverse correlation was observed, with subjects losing about 12% in comprehension for every 10% increase in translation error (R² = 33%).

Limitations and Future Work

The authors acknowledge that the test did not contain true Level 1 documents, and questions about Level 1 elements were constructed within Level 2 and 3 documents. This means the test might not fully capture MT performance at the lowest ILR level. Additionally, genre and difficulty level were not completely independent in this test due to data limitations.

Future work includes creating new Arabic DLPT-star tests with more tailored document selection for comprehension and ensuring texts and tasks align with intended ILR levels. They are also developing a Mandarin Chinese test with similar specifications, aiming for these to be available for a public MT evaluation in 2007. The identification and analysis of "robust" translations are also highlighted as an area for further investigation.

Summary updated

BibTeX
@inproceedings{jones2007ilr,
  author = {Douglas Jones and Martha Herzog and Hussny Ibrahim and Arvind Jairam and Wade Shen and Edward Gibson and Michael Emonts},
  title = {{ILR-Based MT Comprehension Test with Multi-Level Questions}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
}
Figure