TL;DR
The SCATE project aimed to improve the professional translation workflow by developing and integrating advanced technologies for computer-aided translation (CAT). This paper details their work on enhancing fuzzy matching, integrating translation memories with machine translation, improving quality estimation, and developing a more user-friendly interface, ultimately leading to a more efficient and customizable translation environment.
What Problem Does This Paper Address?
Professional translators rely heavily on Computer-Aided Translation (CAT) tools to enhance consistency, productivity, and quality. However, existing CAT tools often present information in a "black box" manner, offering limited insight into the origin or quality of suggestions. This lack of transparency can hinder translator trust and decision-making. The SCATE project addresses this by investigating improvements across various stages of the translation workflow, from core technologies like fuzzy matching and machine translation integration to user interface design and quality estimation, with the goal of creating a more intelligent, transparent, and user-centric translation environment.
Methods
The SCATE project explored several innovative approaches:
- Improved Fuzzy Matching: Beyond traditional string-based metrics, SCATE investigated using linguistic information like lemmas, part-of-speech tags, and syntactic parse trees to improve the accuracy and usefulness of fuzzy matches. They combined multiple metrics using regression trees, showing significant improvements in correlation with evaluation scores and recall.
- TM-MT Integration: To boost MT quality and confidence, SCATE developed a system that integrates Translation Memory (TM) with Machine Translation (MT). This involved "fuzzy match repair" (automatically editing close TM matches) and "span pre-translation" (constraining MT output with TM segments). They tested this with phrase-based SMT and neural MT (NMT) systems.
- Quality Estimation (QE): SCATE developed informative QE systems that go beyond predicting overall post-editing effort. They created a fine-grained MT error taxonomy and used it to build models that not only estimate quality but also provide justifications for the predictions, aiming for "white box" systems. They also explored automatic error detection using RNNs for fluency and accuracy errors.
- Terminology Extraction: The project investigated translators' methods for acquiring domain-specific terminology and developed approaches for automatic term extraction from comparable texts, including using Wikipedia and Reuters news articles. They also explored bilingual lexicon induction (BLI) by combining word and character-level representations.
- Speech Recognition Adaptation: To improve ASR accuracy in translation contexts, SCATE explored adapting language models using information from translation engines and domain-specific data. They also investigated automatic domain adaptation for speech recognition.
- User Interface Design: A significant focus was on creating an "intelligible" user interface that provides clear explanations for translation suggestions (e.g., origin, context, frequency). This was achieved through visual aids and user-centered design, with an emphasis on allowing users to customize the level of detail presented.
Datasets
- DGT Dataset: Used for experiments on TM-MT integration across ten language pairs.
- English-Dutch Comparable Corpus (Medical Domain): Compiled from Wikipedia articles for experiments on terminology extraction and BLI.
- Reuters News Articles Dataset: Used for comparable corpora experiments in terminology extraction.
- Existing Comparable Corpora: From the TTC project, specialized in domains like wind energy and mobile technology.
- VRT Documentaries: A dataset of 14 documentaries used for automatic domain adaptation in speech recognition.
- SCATE MT Error Corpus: A corpus of machine translation errors annotated according to the SCATE taxonomy, used for quality estimation research.
- EMEA TM (English-Dutch): Used for the user study comparing SCATE with Lilt.
Key Insights and Ideas
- Linguistic Information is Key: Beyond simple string matching, incorporating linguistic features (lemmas, syntax) significantly improves fuzzy matching and translation quality.
- TM-MT Synergy: Integrating TM and MT is crucial for building translator trust and improving MT output. Fuzzy match repair and span pre-translation are effective strategies.
- Transparency Builds Trust: Providing translators with clear explanations about the origin and quality of suggestions (e.g., through intelligible visualizations) is vital for user acceptance and efficient decision-making.
- Error Taxonomy for Better QE: A fine-grained error taxonomy is essential for developing informative quality estimation systems that can explain why a translation is good or bad.
- Domain Adaptation is Crucial: For speech recognition in translation, adapting language models to specific domains and topics significantly improves accuracy.
- User-Centric Design Pays Off: Involving professional translators throughout the development process leads to interfaces that better meet their needs and workflows.
Key Results
- Fuzzy Matching: Combined metrics showed a stronger correlation with evaluation scores than baseline methods, with significant improvements in mean evaluation score and recall.
- TM-MT Integration: Achieved significantly higher BLEU scores for nine out of ten language combinations compared to baselines.
- Quality Estimation: Developed informative QE systems that can estimate post-editing time with high accuracy, identifying key error types that drive predictive power. Automatic error detection models achieved marked improvements.
- Speech Recognition: Phrase-based MT models and named entity models reduced recognition errors by approximately 6.5% absolute and 25% relative.
- User Study: Participants found the SCATE interface positive, with intelligible visualisations helping them assess suggestion quality without being distracting. They preferred aids when the information was novel and beneficial.
Limitations and Future Work
- Data Availability: The paper notes the difficulty in finding comparable corpora for bilingual term extraction, which hinders the development of supervised ML approaches.
- Domain Specificity: Some experiments, like the SCATE prototype's MT component, were trained on specific domains (English-Dutch medical texts), limiting generalizability without further adaptation.
- User Preferences: The study highlighted that translators have diverse working styles, and CAT tools need to support these different preferences (e.g., mouse vs. keyboard shortcuts, copying full suggestions vs. building word-by-word).
- Future Work: The authors suggest further research into tree-to-tree approaches for NMT, exploring more advanced semantic information for alignment, and developing more robust methods for punctuation and segmentation insertion in speech translation. They also highlight the need for more efficient web search strategies and better integration of external linguistic resources into CAT tools.
Summary updated