Abstract
Translation ability is known as one of the most difficult language abilities to measure. A typical method of measuring translation ability involves asking translators to translate sentences and to request professional evaluators to grade the translations. It imposes a heavy burden on both translators and evaluators. In this paper, we propose a practical method for assessing translation ability. Our key idea is to incorporate translators' vocabulary knowledge for translation ability assessment. Our method involves just asking translators to tell if they know given words. Using this vocabulary information, we build a probabilistic model to estimate the translators' vocabulary and translation abilities simultaneously. We evaluated our method in a realistic crowdsourcing translation setting in which there is a great need to measure translators' translation ability to select good translators. The results of our experiments show that the proposed method accurately estimates translation ability and selects translators who have sufficient skills in translating a given sentence. We also found that our method significantly reduces the cost of crowd-sourcing translation.
Introduction
The rise of crowdsourcing platforms has made it easy to hire translators, but there is a massive variance in quality among non-professional workers. Traditionally, "vetting" a translator requires a two-sided burden: the translator must perform a test translation, and a professional evaluator must grade it. This is slow, expensive, and often consumes a significant portion of the project's budget before the actual work even begins.
The authors seek a "proxy" for translation ability that is cheap to measure. They identify vocabulary knowledge—a well-established metric in linguistics for general language proficiency—as a candidate. The challenge is building a mathematical framework that can reliably map a translator's simple "Yes/No" responses about word knowledge to their actual ability to produce acceptable sentence-level translations.
Methods
The authors propose the Vocabulary Knowledge-based Translation Quality Predictor (VKTQP). The system is built on two interconnected probabilistic models, as shown in the plate notation below:
- The Translation Ability Model: Based on the Rasch model (a standard in psychometrics), it calculates the probability of a successful translation as a function of the translator’s ability minus the sentence's difficulty. Sentence difficulty is estimated using features like N-gram perplexity and length.
- The Vocabulary Ability Model: Similarly, this predicts if a translator knows a specific word based on their vocabulary "ability" and the word's "difficulty" (derived from frequency in corpora like Google 1-gram and the Brown corpus).
The "secret sauce" is VKTQP-D (Dependent prior). Unlike a standard model that treats these two skills as separate, VKTQP-D uses a mathematical prior that assumes translation ability and vocabulary ability are correlated. To handle the "unreliable narrator" problem (translators who might lie or be overconfident in a self-report test), the model includes an unreliability parameter ($\phi_k$). This allows the system to discount the self-reports of users whose claimed word knowledge doesn't match the expected difficulty patterns.
Experimental Setup
The researchers evaluated their method on an English-to-Japanese translation task using 104 English sentences from Wikipedia’s Kyoto Articles. 55 translators were recruited via Lancers, a major Japanese crowdsourcing platform. Ground truth was established through 1,498 translations manually graded on a 5-point scale by two native Japanese speakers fluent in English. Word difficulty features were pulled from Google 1-gram, COCA, the Brown corpus, and the SVL 12000 difficulty index.
Results and Key Insights
The primary metric used is the Cost Reduction Rate (CRR), which measures how many fewer translation requests are needed to find an "acceptable" translation compared to random selection.
- 32.7% Cost Reduction: In a completely unsupervised setting (zero expert-graded labels), the VKTQP-D model reduced the cost of finding a good translator by nearly a third.
- Statistical Significance: VKTQP-D consistently outperformed standard baselines (like Logistic Regression or SVMs) and even its own "independent" variant, especially when training data was scarce.
- Validation of Correlation: The core hypothesis—that vocabulary knowledge is a high-signal proxy for translation ability—was validated. While translation is a complex creative act, it is fundamentally constrained by lexical access.
As shown in the figure above, the researchers found a clear, statistically significant correlation ($r = 0.941$ in the best-fit model) between estimated translation ability and vocabulary ability. The VKTQP-D model significantly outperformed the VKTQP-I (Independent) model, confirming that assuming a correlation between vocabulary and translation reflects the reality of human linguistic skill.
Conclusion
This study demonstrates that a simple, low-cost vocabulary self-test can effectively predict translation quality, reducing the cost of finding high-quality crowdsourced translators by over 30% without needing initial expert grading. While the study focused on English-to-Japanese, the authors suggest that future work could incorporate other signals, such as editing information, to further refine ability estimates across different language pairs.
Summary updated