TL;DR
This paper presents an evaluation of machine translations of Russian scientific texts into English produced at the National Physical Laboratory (NPL). The study aimed to assess the usefulness of these translations for domain experts and to provide insights into the effectiveness of their translation methods, ultimately guiding potential machine translation service providers.
What Problem Does This Paper Address?
The paper tackles the challenge of assessing the practical utility of machine translation (MT) for specialized scientific content. While MT technology was advancing, its real-world applicability, particularly for translating technical documents, remained a key question. The authors sought to bridge the gap between theoretical MT capabilities and the actual needs of end-users, specifically scientists who would rely on these translations for their work. They recognized that evaluating MT requires considering the context of use, where users often possess domain expertise and can leverage surrounding text to overcome translation imperfections.
Methods
The core of the paper is an evaluation experiment designed to emulate a real-world translation service. Practicing scientists from fields like general physics, electronics, and electrical engineering were invited to submit Russian papers. NPL then provided machine translations of these papers in return for the scientists' feedback on their usefulness. A key aspect of the evaluation was that users were asked to comment on the adequacy of an entire information unit (e.g., a journal article) rather than individual sentences, reflecting how such texts are typically consumed.
To quantify the feedback, a grading scale was devised, ranging from 0 (useless) to 8 (fully adequate), with specific descriptions for each level. This scale was designed to give more weight to the middle range of usefulness, acknowledging that translations might be partially useful even if not perfect. The authors also provide a summary of the translation methods employed, including text preparation, dictionary look-up (using a 18,000-entry Harvard-based dictionary), stem and suffix organization, and a multi-pass Russian analysis algorithm. This algorithm involved stages for determining nominal structures, establishing links between them, identifying conjunctions, creating predicate structures, and handling difficult words.
Datasets
The paper does not explicitly list or introduce new datasets for training MT models. Instead, it focuses on evaluating the output of their existing NPL machine translation system. The "dataset" for evaluation consisted of 44 Russian scientific papers submitted by practicing scientists. Of these, 21 were translated and included in the final evaluation, resulting in 34 comments from evaluators.
Key Insights and Ideas
- Domain Expertise is Crucial: The evaluation strongly suggests that domain experts can derive significant value from machine translations, even if they contain syntactic awkwardness or multiple-meaning choices. Their inherent knowledge helps them comprehend the text.
- Holistic Evaluation Matters: Assessing translations at the level of information units (articles, chapters) rather than isolated sentences provides a more realistic measure of usefulness. Users can often overlook or work around minor translation errors if the overall message is clear.
- A Demand for MT Exists: Despite the limitations, the consensus among evaluators indicated a real demand for translations of this quality, justifying further investment in broader evaluation exercises.
- Translation Challenges Persist: Common criticisms included missing or inappropriate word equivalents, issues with syntactic resolution (ambiguities, wrong resolutions), and word order problems.
- Idioms and Abbreviations are Difficult: The limited size of their idiom list and the challenges in handling abbreviations highlighted areas where MT systems struggled.
- Cost and Speed are Key Drivers: For machine translations to be attractive, they must demonstrate clear advantages in cost and speed over human translations.
Key Results
The evaluation experiment produced 34 comments on 19 papers. The mean usefulness score for the machine translations was 5.6 on a scale of 0 to 8. This score indicates that, on average, the translations were considered "mostly very good" to "fair," with something essential sometimes being lost but generally clear enough. The distribution of scores, visualized in Figure 1, shows a concentration around the higher end of the scale, with a significant number of comments falling into the 6-8 range.
Qualitatively, evaluators provided specific criticisms related to word choice, syntax, and idiomatic expressions. However, some positive comments were also received, with one evaluator finding the translation easier to read with familiarity and another noting that the MT output was less misleading than a published human translation.
Limitations and Future Work
The authors acknowledge several limitations. The evaluation was based on a relatively small sample of papers and evaluators, and the scale used was not based on rigorous scientific principles, though it proved useful in practice. They also note that the "anglicizing" routine for non-dictionary words could be improved.
Future work suggested includes mounting a broader evaluation exercise with a wider range of potential readers to strengthen the verdict on MT usefulness. They also emphasize the need to fit the MT component into an overall translation system that is carefully specified for the translation market, considering cost and speed advantages. The paper also points to further development work on techniques to overcome specific translation faults, detailed in reference 1.
Summary updated