TL;DR
This paper introduces novel "informed masking" strategies for pre-training multimodal machine translation (MMT) models. By strategically masking visual and textual tokens, particularly those related to linguistic nuances like grammatical gender, the models achieve significantly better performance on MMT tasks, setting a new state-of-the-art on the How2 dataset.
What Problem Does This Paper Address?
Pre-trained language models have revolutionized Natural Language Processing (NLP), often relying on Masked Language Modeling (MLM) to learn representations. However, standard MLM randomly masks tokens, which can be inefficient for capturing specific linguistic patterns crucial for tasks like Multimodal Machine Translation (MMT). MMT aims to translate not just text but also incorporate information from other modalities, primarily visual data, to improve translation quality. Existing MMT models, often built on BERT-like architectures, inherit the limitations of random masking, potentially hindering their ability to grasp subtle language features, especially in cross-lingual contexts where languages have different grammatical structures (e.g., gendered nouns and pronouns). This paper addresses this by proposing more intelligent ways to mask tokens during pre-training to better equip MMT models for these challenges.
Methods
The core contribution of this paper lies in three novel masking strategies that move beyond random token selection:
- More Informed Visual Masking: Instead of randomly selecting visual tokens (image features) to mask, this strategy prioritizes masking tokens associated with "people" categories (e.g., "man," "woman," "boy," "girl"). This is achieved by leveraging object detection information from pre-trained models like Faster R-CNN on the Open Images Dataset to identify these specific visual elements.
- More Informed Textual Masking: This approach focuses on masking textual tokens that reveal grammatical gender. Specifically, it prioritizes pronouns like "he/she," "him/her," and "his/hers" (referred to as "Tpronouns"). The model identifies these words within sentences and selectively masks them.
- More Informed Visual and Textual Masking: This strategy combines the previous two, applying both informed visual and informed textual masking simultaneously.
These strategies are applied within the Visual Translation Language Modelling (VTLM) framework, which integrates masked language modeling (MLM) with masked region classification (MRC) to learn cross-lingual and multimodal representations. The VTLM model takes a concatenated input of source language text, target language text, and image features. The overall loss is a combination of MLM and MRC losses. The paper experiments with varying proportions of these informed tokens (e.g., 33.34%, 50.0%, 66.67%) within the standard 15% masking ratio.
Datasets
The paper utilizes the How2 corpus for both pre-training and evaluation. This corpus is a large-scale multimodal and multilingual dataset containing approximately 80,000 instructional videos with English subtitles and crowdsourced Portuguese translations. For pre-training, a subset of 155k features and corresponding text in English and Portuguese was used.
Key Insights and Ideas
- Strategic Masking Matters: The core insight is that how tokens are masked during pre-training significantly impacts downstream performance, especially for nuanced language tasks like MMT. Random masking is a blunt instrument; informed masking allows models to learn specific linguistic and visual patterns more effectively.
- Grammatical Gender is a Key Challenge: The paper highlights how differences in grammatical gender across languages (e.g., English vs. Portuguese) pose a significant challenge for MMT. Models that can better understand and translate gendered pronouns and adjectives will perform better.
- Visual Cues for Linguistic Nuance: Informed visual masking, by focusing on "people" categories, indirectly helps the model learn to associate visual cues with gendered language, improving pronoun translation.
- Diminishing Returns: While informed masking generally improves performance, there's a point of diminishing returns. Over-masking specific types of tokens (e.g., too many "people" visual tokens) can negatively impact the learning of other categories.
Key Results
- State-of-the-Art Performance: The proposed informed masking strategies, particularly when applied to the VTLM model for video subtitles, achieved state-of-the-art accuracy on the Portuguese-English MMT task on the How2 dataset, reaching 52.70 BLEU.
- Significant Improvements: Compared to the baseline random masking VTLM, the informed visual masking strategy showed improvements of up to 1 BLEU and 1.6 METEOR. Informed textual masking also yielded improvements of around 0.7 BLEU and 1.3 METEOR.
- Qualitative Gains: The qualitative analysis demonstrated that the informed masking approaches led to more accurate translations of pronouns (e.g., correctly translating "he/she" instead of "it") and possessive adjectives, addressing common errors made by the baseline. The informed visual masking was particularly effective in correcting pronoun usage related to visual context.
Limitations and Future Work
Limitations:
- Non-Binary Pronouns: The current work primarily addresses binary linguistic forms (masculine/feminine). The authors acknowledge that the improvements did not cover non-binary pronouns (e.g., they/them/theirs), and the underrepresentation of non-binary genders in the data contributes to propagating misrepresentations.
- Limited Scope: Experiments were conducted using VTLM as the base model for the MMT task. While the authors believe the method is generalizable, it hasn't been tested on other models or NLP tasks.
Future Work:
- Investigate Masking Probabilities: Further explore the impact of visual and textual masking probabilities.
- More Effective Masking: Continue to explore more effective masking approaches for downstream MMT performance.
- Fairer Representation: The authors express a hope to contribute to fairer representation of disadvantaged groups, including non-binary genders, in future work.
Summary updated