TL;DR
This paper investigates how the quality of machine translation (MT) output affects human post-editor productivity and behavior. They found that while better MT systems offer a productivity boost of around 20%, individual post-editor differences in work style and skill have a much larger impact.
What Problem Does This Paper Address?
The field of Statistical Machine Translation (SMT) has seen significant progress, leading to increased interest in using MT as a tool to aid human translators through post-editing. While studies have shown productivity gains, there's a crucial open question: how does the quality of the MT output itself influence post-editor efficiency and their editing process? Furthermore, it's unclear if certain MT approaches (like syntax-based models) are inherently better suited for post-editing than others. This paper aims to shed light on these questions by directly comparing the impact of different SMT systems on post-editing effort and behavior.
Methods
The researchers conducted an experiment involving four volunteers who post-edited English-to-German news translations generated by four different SMT systems. To isolate the effect of the MT system, they used a balanced experimental design where each post-editor worked on an equal amount of output from each system, and each sentence was post-edited once by one of the four post-editors. This random assignment helps to mitigate the influence of individual post-editor differences when analyzing system performance.
Productivity was measured by the time spent on each sentence. Beyond raw speed, they also analyzed editing actions by tracking character provenance (whether a character came from the original MT, was typed, or was pasted) and token provenance (including partially edited words). They also logged various editing activities like pauses and working time to understand the post-editing process in more detail.
Datasets
The study utilized news stories from the WMT 2013 evaluation campaign. Specifically, they selected 9 different news stories, totaling 500 sentences, originally written in English. The MT output for these stories was generated by four different SMT systems that participated in the WMT 2013 evaluation. The paper notes that this data, along with the MT output, is available for download.
Key Insights and Ideas
- Post-editor variance is king: The most striking finding is the significant variability in post-editing styles and speeds among the individual post-editors. This variance dwarfs the differences observed due to the MT system quality.
- MT quality matters, but less than expected: While better MT systems do lead to a productivity gain (around 20%), the differences between the top three systems were not statistically significant. The lowest-performing system did show a noticeable decrease in productivity.
- Syntax-based MT's subtle advantage: The syntax-based system (UEDIN-SYNTAX) showed a slight edge in post-editing speed and a lower HTER (Human Translation Edit Rate), suggesting potential benefits in producing more linguistically sound output that requires less drastic editing, even if automatic metrics don't always fully capture this.
- Editing actions reveal process: Analyzing character and token provenance provided insights into how post-editors interact with MT output. For instance, while cutting and pasting were observed, it was less common than expected, with many "moved" words being deleted and retyped.
- Pauses are key indicators: The breakdown of editing activities revealed that longer pauses (especially "big pauses" over 60 seconds) were strongly correlated with lower quality MT output and indicated difficulties in understanding and translating complex segments.
Key Results
- Productivity Gain: The best MT systems led to approximately a 20% productivity gain compared to the worst system.
- System Performance: The system UU (a phrase-based system trained on smaller corpora) was significantly worse than the others, showing about 20% lower productivity. The other three systems (ONLINE-B, UEDIN-PHRASE, UEDIN-SYNTAX) performed similarly.
- Post-Editor Differences: Post-editor 3 took over three times longer than the fastest post-editor (Post-Editor 1).
- HTER Scores: The HTER scores showed similar trends to productivity, with UU lagging behind. Notably, ONLINE-B had a statistically better HTER score than UEDIN-PHRASE.
- Character Provenance: The fastest post-editor (PE 1) left the most characters unchanged (72.9%), while the slowest (PE 3) used significantly more cutting and pasting.
Limitations and Future Work
The authors acknowledge that their study used only four post-editors, who were volunteers and not professional translators. They suggest that future work should validate these findings with professional post-editors. The small sample size also limits the statistical significance of some findings, particularly concerning the subtle differences between systems and post-editors. They also highlight the need for further research to gain deeper insights into the relationship between MT system quality, editing activities, and post-editor behavior, especially concerning the impact of pauses and the benefits of syntax-based MT.
Summary updated