Introduction
In automated language learning, providing a single "gold standard" translation is often insufficient. Students may provide various valid synonyms or grammatical structures that a rigid system might incorrectly penalize. The STAPLE (Simultaneous Translation And Paraphrase for Language Education) shared task at WNGT 2020 addressed this by challenging participants to generate the exhaustive set of acceptable translations for English prompts across five languages: Hungarian, Japanese, Korean, Vietnamese, and Portuguese. Researchers from Charles University (CUNI) and LMU Munich developed an "Expand and Filter" approach to maximize translation recall while maintaining high precision.
The Expand and Filter Pipeline
The proposed system operates in two distinct stages: candidate generation (Expand) and quality pruning (Filter).
1. The Expand Phase
The system first generates a broad set of candidate translations using a standard Transformer model implemented in the Marian toolkit. Rather than selecting only the top-ranked translation, the team extracted n-best lists ranging from 50 to 1000 candidates. To further diversify the output, they experimented with a Levenshtein Transformer (LevT). Unlike traditional autoregressive models, the LevT refines text through iterative deletion and insertion operations, allowing it to "hop" between different paraphrastic forms of a sentence.
2. The Filter Phase
Generating a massive volume of candidates inevitably introduces ungrammatical or semantically incorrect "noise." To mitigate this, the researchers employed a cross-lingual filtering classifier based on XLM-RoBERTa. This model evaluates (Source, Candidate) pairs and assigns a probability of correctness. Only candidates exceeding a specific threshold ($C_{thr}$) are retained in the final output set.
Data Selection and Training
The team utilized the provided STAPLE dataset alongside external parallel corpora from OPUS (e.g., ParaCrawl, JParaCrawl) and in-domain data from Tatoeba. A key finding was the "In-Domain Advantage": smaller, high-quality datasets like Tatoeba often yielded better results for the specific pedagogical style of Duolingo prompts than massive, noisy general-purpose datasets. Monolingual data from Wikipedia and NewsCrawl was also leveraged via back-translation to augment the training sets.
Key Insights and Results
The "Expand and Filter" strategy proved highly effective, outperforming the best baseline models by 10–22% in absolute weighted F1 score. The system saw its most significant gains in Portuguese.
A critical insight involved the relationship between beam search and filtering. In standard NMT, increasing the beam size typically causes precision to plummet as the model explores lower-probability (and often lower-quality) paths. However, the CUNI-LMU team demonstrated that a robust filtering layer allows the system to exploit larger beam sizes to capture rare but valid translations without the usual precision penalty.
The Paraphrase Paradox and Future Work
Despite its theoretical appeal, the Levenshtein Transformer introduced significant noise that the classifier struggled to filter perfectly. This "Paraphrase Paradox" suggests that while diversity is necessary for high recall, unconstrained generation can easily lead to ungrammatical traps. Future research will focus on tighter integration between the generation and filtering components, potentially using the classifier's feedback to guide the paraphrasing process during training.
Summary updated