TL;DR
This paper introduces a new evaluation metric, ISDIT, and a specialized corpus to better assess the "focality" of translations for Japanese statutes that have been partially amended. This is crucial for ensuring that only the relevant parts of a statute's translation are updated, maintaining accuracy and avoiding confusion for international readers.
What Problem Does This Paper Address?
Governments worldwide need to quickly update their statutes to facilitate international trade and investment. Japan's government launched a translation system (JLT) for its statutes, but a significant portion of translations are outdated. When a statute is amended, especially partially, its English translation needs to be updated promptly. The challenge lies in "focal translation": modifying only the expressions directly affected by the amendment while keeping the rest of the translation intact. This ensures that the translation accurately reflects the changes and leverages the reliability of existing, unchanged translations. Previous methods and evaluation metrics for this task were found to be insufficient, either not adequately assessing the quality of translated changed expressions or using datasets that contained non-focal examples.
Methods
The paper proposes a new evaluation metric called the Inclusive Score for DIfferential Translation (ISDIT). Unlike previous metrics that focused solely on retaining unchanged expressions, ISDIT incorporates both n-gram recall of unchanged expressions and n-gram precision of the system's output against a reference. This dual approach aims to simultaneously evaluate the translation quality of both the changed and unchanged parts of the statute.
The authors also introduce a focal differential translation corpus. This corpus was compiled by carefully selecting statutes where JLT versions lagged behind the most recent e-LAWS versions. Crucially, the compilation process involved human translators to ensure that the post-amendment translations in the corpus strictly adhere to the focality requirement, meaning only amendment-relevant expressions were modified.
Datasets
The paper utilizes and introduces the following datasets:
- Japanese Law Translation Database System (JLT): Used as a source for existing statute translations.
- e-LAWS: A Japanese government database for statutes, used in conjunction with JLT to identify statutes with outdated translations.
- Focal Differential Translation Corpus (New): A newly compiled corpus of 1,483 differential translation examples from 62 amendment cases. This corpus was created by human translators to ensure strict focality. It includes phrase-level and sentence-level modifications.
- Kozakai et al. (2017) corpus: Used for training data, consisting of 158,928 Japanese-English sentence pairs.
- Combined corpus for training: The authors mixed the Kozakai et al. corpus with an additional corpus compiled from JLT statutes, resulting in 232,830 sentence pairs from 462 statutes.
Key Insights and Ideas
- Focality is paramount: Simply translating the amended statute without considering which parts were actually changed leads to confusion and inaccurate legal information for international readers.
- Existing metrics are insufficient: Metrics like BLEU and RIBES, while useful for general translation quality, do not specifically measure the "focality" of translations for amended statutes. Previous focality metrics also had limitations.
- ISDIT offers a more holistic evaluation: By combining recall of unchanged parts with precision of changed parts, ISDIT provides a more comprehensive assessment of focal translation quality.
- Human-curated datasets are essential: Automatically generated or existing datasets may not guarantee focality. A corpus specifically designed and verified by human translators is crucial for reliable evaluation.
- NMT with template-aware SMT shows promise: Combining Transformer-based NMT with template-aware SMT, especially with techniques like MC dropout, appears to yield good results for both general translation quality and focality.
Key Results
- The proposed ISDIT metric showed a high correlation (0.927) with the previously proposed focality scores, indicating it captures similar aspects of focality.
- The Transformer + Kozakai model + MC dropout method achieved the best performance in BLEU and RIBES scores among the evaluated models.
- Naive template-aware SMT methods performed best on the focality scores, but their overall translation quality (BLEU, RIBES) was lower.
- The authors' proposed method (Transformer + WPoT + MC dropout) achieved the highest ISDIT score (81.95) and a strong focality score (90.92), outperforming previous methods.
- Qualitative analysis revealed that methods like Kozakai's struggled with translating specific terms (e.g., "協会" as "NHK") when they appeared in a broader context of amendment, whereas the proposed method handled this better by considering the whole sentence.
Limitations and Future Work
- Corpus size: The authors acknowledge that the size of their newly compiled focal corpus (1,483 examples) could be increased for better neural network training.
- ISDIT weighting: The optimal weighting of the two factors (recall and precision) within the ISDIT metric needs further investigation.
- Domain expansion: The authors suggest exploring the application of ISDIT to other domains involving version-controlled documents, such as contracts and technical manuals.
- Further improvements to NMT: While NMT combined with template-aware SMT showed good results, there is always room for further improvements in capturing the nuances of focal translation.
Summary updated