TL;DR
This paper introduces an automated method to improve machine translation (MT) rules by leveraging online post-editing feedback from bilingual speakers. By analyzing user corrections, the system can refine its grammar and lexicon, leading to more accurate and generalizable translations, effectively turning the internet into a collaborative MT improvement tool.
What Problem Does This Paper Address?
Achieving high-quality machine translation remains a significant challenge. While researchers have explored incorporating user feedback, most systems struggle to effectively integrate post-editing efforts beyond simply adding corrected translations to training data. This paper addresses the gap of automatically refining MT rules based on user-provided corrections, aiming to improve translation quality in a scalable and efficient manner, especially for resource-poor languages where expert linguists are scarce.
Methods
The core of the proposed method is an automated Rule Refinement (RR) module that processes feedback from an online Translation Correction Tool (TCTool).
- Elicitation: The TCTool allows bilingual speakers to correct MT output by adding, deleting, or modifying words and alignments, and even reordering them. This minimal post-editing information is crucial for the system.
- Error Extraction: User corrections are logged, and the RR module extracts relevant information, creating a "Correction Instance."
- Lexical Refinement: The RR module first checks the lexicon. If a correction involves a new word sense (e.g., "great" translating to "gran" in Spanish), it adds a new lexical entry. This is done by duplicating an existing entry and adding a distinguishing feature (e.g.,
feat1=+for pre-nominal adjectives). - Grammar Refinement: The RR module then analyzes the MT system's parse trace to identify the grammar rule responsible for the error. It can either directly modify the original rule or, more commonly, bifurcate it by creating a copy and applying the refinement to the copy. This preserves the original rule's functionality while allowing for specific adaptations. For instance, a rule for noun phrases might be modified to handle pre-nominal adjectives correctly.
- Generalization: The refined rules are designed to generalize, meaning they not only fix the specific sentence corrected by the user but also improve translations of similar sentences.
Datasets
Not explicitly detailed in the paper, but the system relies on:
- Parallel corpora: Used for initial rule learning.
- Elicitation corpus: Sentences produced by the MT system that are then corrected by users via the TCTool.
- User studies: Conducted to determine the optimal amount of error information to elicit from non-expert bilingual speakers.
Key Insights and Ideas
- Leveraging the Internet for MT Improvement: The paper proposes a novel approach to harness the collective power of the internet and bilingual speakers to collaboratively improve MT systems, making it a "game with a purpose."
- Automated Rule Refinement: The core innovation is automating the complex process of identifying and fixing incorrect translation rules based on user feedback, reducing reliance on expert linguists.
- Bifurcation as a Refinement Strategy: Creating a copy of a grammar rule and refining the copy is a robust way to adapt rules without breaking existing functionality, especially when dealing with nuanced linguistic phenomena like adjective placement.
- Generalization from Specific Corrections: The system's ability to generalize corrections beyond the exact sentence corrected is key to its effectiveness, leading to broader improvements in translation quality.
- Addressing Resource-Poor Scenarios: This method is particularly valuable for languages with limited linguistic resources, where traditional MT development is challenging.
Key Results
- The system successfully refined a Spanish grammar rule to correctly translate "Gaudi was a great artist" to "Gaudí era un gran artista," fixing both lexical (grande -> gran) and word order (adjective placement) errors.
- The refined rule also correctly translated similar sentences, such as "Irina is a great friend" to "Irina es una gran amiga," demonstrating generalization power.
- User studies showed that simple error information can be elicited reliably (F1 0.89) compared to more complex error types.
Limitations and Future Work
- Order Dependency: The Rule Refinement process is currently dependent on the order in which user corrections are processed. The authors suggest exploring methods to rank Correction Instances to maximize accuracy in batch mode.
- Rule Dependencies: Detecting and managing dependencies between rules before refinement could lead to more optimal rankings and higher accuracy.
- Interactive Refinement: Future work includes enhancing the system with Active Learning to allow for more interactive user input, asking users for clarification only when necessary to determine the most robust refinement operations. This aims to find an optimal balance between batch and fully interactive systems.
- Evaluation Metrics: Developing comprehensive evaluation metrics that consider not only the presence of correct translations but also the prevention of incorrect ones and the tightening of the grammar (reduced lattice size) is an ongoing area.
Summary updated