TL;DR
This paper investigates how English and Croatian handle discourse phenomena, like coreference, and how well machine translation systems (SMT and NMT) translate them. They found that while NMT generally outperforms SMT, both struggle with complex discourse structures, highlighting the need for more discourse-aware MT.
What Problem Does This Paper Address?
Machine translation (MT) systems often struggle to produce fluent and coherent translations because they don't always capture the nuances of how languages connect ideas across sentences. This is particularly true for "discourse phenomena," which are linguistic devices that create links between different parts of a text, ensuring it flows logically. Coreference, where different words or phrases refer to the same entity (e.g., "John" and "he"), is a prime example. English and Croatian, being structurally very different (English being analytic and Croatian being synthetic and morphologically rich), are likely to express these coreference relations differently. This divergence poses a significant challenge for MT systems, as a direct word-for-word translation might miss the intended connection, leading to unintelligible or awkward output. The paper aims to shed light on these differences and evaluate how well current MT systems handle them.
Methods
The researchers employed a corpus-based approach, extending a framework developed by Lapshinova-Koltunski and Hardmeier (2017). They automatically identified specific discourse patterns in parallel English-Croatian corpora using linguistic annotations (like part-of-speech tags and dependency parsing) and word alignment tools. This allowed them to pinpoint instances where a discourse phenomenon in one language either lacked a direct equivalent in the other or was handled differently.
They then created a specialized test set containing challenging discourse constructions identified in human translations. This dataset was used to evaluate the performance of three MT systems: one statistical machine translation (SMT) system and two neural machine translation (NMT) systems. A detailed, fine-grained error classification scheme was developed to assess the quality of translations, assigning penalties based on the severity of the error, particularly focusing on whether the coreference link was preserved.
Datasets
The study utilized three parallel corpora, each representing a different register:
- DGT-TM: EU legal texts (950K sentences) - formal and standardized.
- SETIMES2: Newspaper articles (200K sentences) - relatively informal.
- TedTalks: Prepared speeches (86K sentences) - informal and diverse.
The paper also introduced a newly constructed test dataset of 1899 sentence pairs specifically designed to include challenging discourse phenomena. This dataset has been made publicly available for future research.
Key Insights and Ideas
- Language Divergence is Key: The significant structural differences between English and Croatian (e.g., English articles vs. Croatian demonstratives/possessives, Croatian pro-drop nature) create inherent challenges for translating discourse.
- NMT Shows Promise, But Not Perfection: While NMT systems generally outperformed SMT, they still struggled with complex discourse phenomena, especially those requiring creative paraphrasing or handling lexical gaps.
- Register Matters: The performance of MT systems varied across different text types, with systems generally performing better on the more formal and repetitive DGT corpus compared to the informal TedTalks corpus.
- Unaligned Phenomena are a Hurdle: Phenomena that don't have a direct one-to-one equivalent in the target language (e.g., certain uses of "koji" in Croatian) proved particularly difficult for MT systems.
- Beyond BLEU: The paper highlights the limitations of standard MT evaluation metrics like BLEU for assessing performance on specific linguistic phenomena, emphasizing the need for more targeted error analysis.
Key Results
- Overall Performance: All MT systems performed relatively poorly on the examined discourse phenomena, with a higher number of unacceptable than acceptable translations.
- SMT vs. NMT: NMT systems generally produced more acceptable translations than SMT, particularly on unaligned phenomena.
- Corpus Performance: Systems performed best on the DGT corpus and worst on the TedTalks corpus. SMT showed the largest variance in performance across corpora.
- Specific Phenomena:
- NMT systems outperformed SMT on all three unaligned phenomena.
- SMT performed worst on possessives, while NMT systems did better at identifying contexts where they could be omitted.
- Both NMT systems showed good performance on "it" as an expletive or passive subject, which often requires rephrasing in Croatian.
- All systems struggled with creative paraphrasing, lexical gaps, and idiomatic expressions.
Limitations and Future Work
- Limitations: The study primarily focused on intra-sentential phenomena due to the nature of automatic extraction. The artificially constructed test set, while necessary for testing a variety of phenomena, might be considered a constraint.
- Future Work: The authors suggest investigating other coreference phenomena, exploring different linguistic features for extraction patterns, and experimenting with more advanced MT architectures like the Transformer. They also propose integrating coreference annotation systems into MT workflows to improve discourse awareness. The insights gained from English-Croatian could potentially be applied to other closely related Slavic languages.
Summary updated