TL;DR
This paper presents a system for simultaneous translation and paraphrasing, achieving top performance in a competition for language education. By fine-tuning pre-trained Transformer models with a novel oversampling strategy and a simple post-processing step, they significantly improved performance over baselines on Hungarian and Portuguese.
What Problem Does This Paper Address?
The paper tackles the challenge of generating multiple valid translations (paraphrases) for a given English sentence into a target language, specifically for language education purposes. This is crucial for developing automated grading and curation systems for language learners' responses. The core problem lies in balancing the precision of translations with the coverage of diverse, acceptable paraphrases, especially for languages with limited resources. Existing work often focuses on either translation or paraphrasing, but not the simultaneous generation of multiple paraphrased translations.
Methods
The authors' approach is built upon pre-trained Transformer models, a standard in modern NLP. The key innovations lie in:
- Fine-tuning: They fine-tune existing pre-trained Transformer models (from OPUS-MT) on the specific task data.
- Weighted Sampling: A novel oversampling strategy is employed where translation pairs are duplicated based on their learner response frequency (weights). This effectively prioritizes more common or accepted translations during training, while filtering out very infrequent ones.
- Beam Search: A wide beam search is used during decoding to generate multiple hypotheses. The beam size is tailored to each language track (10 for Hungarian, 28 for Portuguese).
- Post-processing: A simple yet effective post-processing step removes translation predictions where the maximum token-level score falls below a certain threshold (-3.5), determined through validation experiments.
Datasets
The primary dataset for this work comes from a shared task hosted by Duolingo. It consists of:
- Training Data: 4000 English prompts for both Hungarian and Portuguese, leading to 251,442 English-Hungarian (en-hu) and 526,466 English-Portuguese (en-pt) translation pairs. These pairs are accompanied by learner response frequencies, used for weighting.
- Development and Test Data: 500 prompts for each language in both dev and test sets.
- Baseline Data: Translations from AWS were also provided as a baseline.
Key Insights and Ideas
- Oversampling is Key: The oversampling strategy, which prioritizes translations based on learner frequency, provided a substantial improvement (around 10.7 absolute points in Weighted Macro F1) over a baseline without it. This highlights the importance of incorporating real-world usage data.
- Simple Post-processing Works Wonders: A straightforward thresholding of token-level scores during post-processing yielded a noticeable gain (around 1.79 absolute points), demonstrating that even simple heuristics can be effective.
- Fine-tuning Pre-trained Models is Powerful: The significant improvement from fine-tuning pre-trained models (around 15.4 points) underscores the value of leveraging large-scale pre-trained language models for downstream tasks.
- Oversampling + Fine-tuning + Post-processing is a Winning Combo: Interestingly, despite exploring more complex variants like multi-output sequence formulation and nucleus sampling, the authors found that their final submission strategy (oversampling, fine-tuning, and post-processing) remained superior.
Key Results
The proposed system achieved significant improvements over the strong AWS baseline:
- Hungarian (en-hu): ~30 absolute points improvement on the dev set and ~27 points on the test set in Weighted Macro F1.
- Portuguese (en-pt): ~34 absolute points improvement on the dev set and ~33 points on the test set in Weighted Macro F1.
The system ranked 1st on the dev leaderboard and 2nd on the test leaderboard for the en-hu track, and 2nd on the dev leaderboard and 3rd on the test leaderboard for the en-pt track.
Limitations and Future Work
The authors acknowledge several areas for future exploration:
- Advanced Pre-trained Models: Investigating the use of newer state-of-the-art pre-trained models like BART and T5 for initialization could potentially boost performance.
- Cross-lingual Models: Exploring cross-lingual models such as XLM-Roberta, potentially by initializing the encoder with their representations, is a promising direction.
- Joint Training: Concatenating training datasets from multiple language pairs and training a joint model could yield additional benefits.
- Model-based Prediction Filtering: While their final submission outperformed this approach, the authors did explore using an XGBoost classifier for filtering predictions, which showed some promise.
Summary updated