TL;DR
This paper tackles the challenge of creating a Russian data-to-text (D2T) generation corpus by leveraging neural machine translation (NMT) and post-editing. The authors introduce a semi-automatic method to translate an existing English D2T corpus into Russian, analyze the resulting errors, and develop techniques to automatically correct them, with a particular focus on named entities.
What Problem Does This Paper Address?
Data-to-text (D2T) generation, the task of transforming structured data into natural language, is a crucial area in Natural Language Generation (NLG). However, building high-quality training data for D2T models is a significant bottleneck. Naturally occurring parallel D2T data is scarce, and manual creation is labor-intensive and complex. Furthermore, existing D2T corpora are predominantly for English, leaving a gap for other languages like Russian. This paper aims to bridge this gap by creating a Russian D2T corpus, enabling the development of Russian D2T models.
Methods
The core of the paper's methodology involves a three-step process:
- Neural Machine Translation (NMT): The authors use a state-of-the-art English-to-Russian NMT system (from the University of Edinburgh, a top performer in WMT17) to translate the English WebNLG D2T corpus into Russian. To handle the sentence-to-sentence nature of the NMT system, they split the WebNLG texts into individual sentences, translate them, and then re-concatenate the translations.
- Error Analysis: A detailed manual error analysis is performed on the NMT output. This analysis focuses on identifying and categorizing translation errors, with a specific emphasis on named entities (NEs), which are found to be a major source of errors.
- Automatic Post-Editing (APE): Two techniques are employed for automatic post-editing:
- Rule-Based Post-Editing: This method extracts rules from manual corrections of single-triple WebNLG instances. It identifies sequences of edit operations (substitution, deletion, insertion) needed to transform the NMT output into a corrected version, effectively learning from edit distance operations.
- Neural Automatic Post-Editing Model: A neural APE model is trained on pairs of NMT outputs and their rule-based post-edited versions. This model aims to learn more generalizable correction patterns.
Datasets
- WebNLG D2T Corpus: This is the primary dataset used. It consists of RDF triples and their corresponding English textual descriptions, extracted from DBpedia. The corpus covers fifteen DBpedia categories and has a hierarchical structure, allowing for the propagation of edits.
- English-Russian NMT System Training Data: The NMT system used for translation was trained on extensive sentence-to-sentence data for the English-Russian News translation task.
- WebSplit: Used for sentence splitting of the WebNLG texts before translation.
- Manual Post-Edited Data: A subset of the translated WebNLG corpus was manually post-edited and annotated for error analysis. This data was also used to train the neural APE model.
Key Insights and Ideas
- Named Entities are the Achilles' Heel: The error analysis reveals that a staggering 53% of all translation errors in the NMT output are related to named entities. This highlights the particular difficulty NMT systems face with proper nouns, locations, and other named entities, especially when dealing with cross-lingual transfer.
- Rule-Based Post-Editing as a Strong Baseline: The rule-based post-editing method, derived from edit distance operations, proved effective in generating corrected data. This approach provides a solid foundation for further automatic correction and serves as a strong baseline for neural APE models.
- Neural APE Struggles to Surpass Rule-Based: Interestingly, the neural APE model, despite being trained on the rule-based post-edited data, did not significantly outperform the rule-based method or even a "do-nothing" baseline (keeping the MT output as is) in terms of BLEU scores. This suggests that the task of correcting NMT output, especially with a strong rule-based baseline, is challenging.
- Hierarchical Corpus Structure is Advantageous: The hierarchical nature of the WebNLG corpus, where descriptions of single triples are available alongside descriptions of triple sets, is beneficial. It allows for the potential to propagate post-edits from simpler to more complex verbalizations.
Key Results
- High Error Rate: Approximately 80% of the automatically translated sentences from English to Russian contained errors.
- NE Dominance: 53% of these errors were attributed to named entities.
- Rule-Based Post-Editing Impact: The rule-based post-editing method increased the amount of post-edited data to 4,188 instances.
- Neural APE Performance: The neural APE model achieved BLEU-4 scores of 83.00 on the development set and 83.65 on the test set, which were not statistically significantly better than the "do-nothing" baseline (81.25 dev, 82.85 test).
- Inter-annotator Agreement: High agreement (Cohen's kappa of 0.67) was observed for the "Named entity" error category, indicating consistency in identifying and correcting these errors.
Limitations and Future Work
- Focus on Named Entities: The current work primarily addresses named entity errors, leaving other error types for future research.
- Neural APE Limitations: The neural APE model did not significantly improve over the baseline, suggesting that further research is needed to enhance its performance, potentially by exploring different architectures or training strategies.
- Generalizing Post-Editing: The authors plan to extend their approach to other error types and investigate how the neural model can be improved to generalize post-editing to errors not covered by the rule-based method.
- Pre-translation NE Identification: A promising future direction is to identify and translate named entities separately before the main translation phase, potentially using a delexicalized version of the corpus. This could mitigate NE-related errors.
- Grammatical Adaptation: The grammatical adaptation of named entities requires more careful investigation.
Summary updated