TL;DR
This paper introduces a specialized test suite to evaluate how well English-to-German machine translation (MT) systems handle pronoun translation, a notoriously difficult aspect of natural language processing. The findings reveal that while modern neural MT systems excel at translating pronouns with clear, within-sentence references, they still struggle with inter-sentence pronoun resolution, highlighting a key area for future research.
What Problem Does This Paper Address?
Machine translation (MT) systems have made significant strides, particularly with the advent of neural MT (NMT). However, a persistent challenge lies in their ability to correctly translate elements that rely on understanding broader discourse context, rather than just immediate word neighbors. Pronoun translation is a prime example of this difficulty. Pronouns like "it" and "they" can refer to entities, events, or even abstract concepts, and their correct translation into German often depends on identifying their antecedents, which may appear in previous sentences. Traditional automatic evaluation metrics, like BLEU, are often too coarse-grained to capture these subtle errors, as they are heavily influenced by word frequency and may overlook specific linguistic phenomena that occur less often. This paper aims to address this gap by providing a more granular evaluation of pronoun translation performance in English-to-German MT systems, specifically within the context of the WMT 2018 shared task.
Methods
The core of this paper is the development and application of a semi-automatic evaluation methodology using a specialized test suite called PROTEST, adapted for English-German. The PROTEST test suite is designed to assess pronoun translation across various categories, including:
- Pronoun Function: Distinguishing between anaphoric (referring to a preceding noun phrase or event), pleonastic (grammatically necessary but semantically empty, like "it" in "it is raining"), and event reference pronouns.
- Antecedent Location: Differentiating between intra-sentential (antecedent within the same sentence) and inter-sentential (antecedent in a different sentence) references.
- Antecedent Type: Considering group nouns as antecedents.
- Pronoun Position: Analyzing subject vs. non-subject positions for "it."
- Special Cases: Examining the use of "they" as a singular pronoun.
The evaluation process involved two main steps:
- Automatic Matching: Candidate translations were automatically matched against reference translations. This step aimed to identify correct translations with high precision.
- Manual Annotation: Examples not automatically approved were sent for manual annotation by trained translators. This step ensured a more comprehensive and accurate assessment, especially for ambiguous cases.
The authors also employed automatic metrics like BLEU and APT (Automatic Pronoun Translation) for a baseline comparison, but emphasized the limitations of these metrics for nuanced pronoun evaluation.
Datasets
The primary dataset used for constructing the test suite was the ParCorFull corpus. This corpus is a German-English parallel corpus annotated for co-reference, containing both nominal and event antecedents. The authors specifically extracted examples from the TED talks section of ParCorFull.
The test suite itself consists of 200 pronoun translation examples for English-German, focusing on the ambiguous pronouns "it" and "they."
For evaluation, the authors used the output of 16 English-to-German MT systems that participated in the WMT 2018 competition, including 10 shared task submissions and 6 anonymized online translation systems.
Key Insights and Ideas
- NMT's Strength in Intra-Sentential References: Modern NMT systems, particularly those based on the Transformer architecture, demonstrate a strong capability in translating pronouns when their antecedents are within the same sentence.
- Inter-Sentential References Remain a Hurdle: The most significant challenge for current NMT systems lies in accurately translating pronouns whose antecedents are in preceding sentences. This highlights a gap in their ability to maintain long-range discourse coherence.
- Event Pronouns are Well-Handled: Unlike older phrase-based SMT systems, NMT systems generally perform well on translating event pronouns, suggesting an improved understanding of abstract references.
- Limitations of Automatic Metrics: The study reinforces the idea that while automatic metrics like BLEU and APT correlate with manual evaluation, they can provide a misleading picture of system performance, especially for specific linguistic phenomena like pronoun translation. They can miss significant outliers and fail to differentiate systems that perform similarly on average but differ in their handling of specific error types.
- No Single System Dominates: The evaluation revealed that no single system excelled at translating all types of anaphoric pronouns, suggesting that random effects or architectural differences still play a role in performance.
Key Results
- The Microsoft Marian system achieved the highest score, correctly translating 157 out of 200 pronouns.
- A group of 5 shared task submissions and 3 online systems also performed well, scoring between 145 and 148 correct translations.
- The automatic evaluation of the PROTEST test suite showed that 86.6% of automatically approved examples were accepted as correct by human annotators, though this accuracy varied significantly by category.
- For intra-sentential anaphoric "it" pronouns, the best systems achieved correct translations for 70-80% of examples.
- In contrast, for inter-sentential "it" categories, the average performance was below 50%, even for the best systems.
- Plonastic "it" pronouns were handled almost perfectly by most systems.
- Plural "they" pronouns were generally translated well using the German "sie," with errors often stemming from confusion with the polite address form.
Limitations and Future Work
- Positive Bias in Evaluation: The semi-automatic evaluation method, by design, tends to approve correct examples rather than reject incorrect ones. This leads to a positive bias, potentially overestimating system performance.
- Small Test Suite Size: The authors acknowledge that the test suite, while balanced, is relatively small. This, combined with variations in system architectures, means that differences in scores should be interpreted with caution.
- Future Work: The paper suggests that future evaluations should focus more on challenging cases, particularly inter-sentential anaphoric pronouns, as these remain a significant challenge for NMT. The authors see these cases as ideal testbeds for developing more context-aware NMT systems, citing recent research in this direction. They also propose that the PROTEST test suite could be further refined and expanded.
Summary updated