TL;DR
This paper introduces "pragmatic masking" and "surrogate fine-tuning" to improve how masked language models (MLMs) like BERT understand social nuances in text. By focusing pre-training on social cues (like hashtags and emojis) and using them for intermediate training, they achieve state-of-the-art results on social meaning tasks, especially in low-data scenarios.
What Problem Does This Paper Address?
Masked Language Models (MLMs) are powerful tools for NLP, but their pre-training objective (predicting masked words) doesn't perfectly align with the goals of downstream tasks, especially those involving subtle social meaning. Fine-tuning MLMs on specific tasks requires a lot of labeled data, which is often scarce for understanding complex social cues like sarcasm, irony, or emotion. The authors aim to bridge this gap by making the pre-training process more relevant to social meaning tasks and by developing a method that can leverage readily available social cues as a form of "distant supervision" to improve model performance, particularly when labeled data is limited.
Methods
The paper proposes two core techniques:
- Pragmatic Masking: Instead of randomly masking tokens during pre-training, this method prioritizes masking social cues like hashtags and emojis. The hypothesis is that by forcing the model to reconstruct these cues, it will learn representations that are more attuned to social meaning. This masking can be done in a self-supervised manner as these cues are easily identifiable.
- Surrogate Fine-Tuning (SFT): This is an intermediate training step that doesn't rely on gold-standard labels for specific downstream tasks. Instead, it uses readily available social cues (like hashtags or emojis appearing at the end of tweets) as "surrogate labels." The model is trained to predict these cues, building a richer understanding of social concepts that can then be transferred to various social meaning tasks.
Datasets
The authors created a social meaning benchmark by collecting 15 datasets from various research sources, covering tasks like:
- Crisis awareness
- Emotion detection
- Hateful and offensive language detection
- Humor detection
- Irony detection
- Sarcasm detection
- Sentiment analysis
- Stance detection
They also created a large pre-training dataset called TweetEng from 2.4 billion English tweets. For SFT, they specifically curated Hashtag_pred and Emoji_pred datasets derived from TweetEng, focusing on hashtags and emojis at the end of tweets.
Key Insights and Ideas
- Social Cues are Powerful: Hashtags and emojis, often seen as informal or noisy, are actually rich sources of information for understanding social meaning.
- Pre-training Matters for Social Tasks: Aligning the pre-training objective with social cues significantly boosts performance on downstream social meaning tasks, even more so than domain-specific pre-training.
- Few-Shot Learning Breakthrough: The proposed methods are remarkably effective in few-shot and severely few-shot settings (as little as 1% of training data), drastically reducing the need for extensive labeled datasets.
- Language Agnosticism: The methods show promise in zero-shot settings for languages other than English, suggesting a generalizable approach.
- Emoji vs. Hashtag: Emojis appear to be slightly more helpful than hashtags for this type of guided pre-training in social meaning tasks, suggesting they carry more direct social meaning signals.
Key Results
- The proposed methods achieved 2.34% F₁ improvement over a competitive baseline on average across 15 social meaning tasks.
- In severely few-shot settings (1% of training data), their models enabled an impressive 68.54% average F₁.
- They achieved new state-of-the-art (SOTA) on eight out of 15 supervised datasets.
- Their models outperformed the domain-specific BERTweet baseline, especially in low-data scenarios.
- In zero-shot settings on non-English datasets, their models showed an average improvement of 5.96 F₁ over the baseline.
Limitations and Future Work
- Uniformity-Tolerance Dilemma: While their models improve uniformity (making embeddings more spread out and distinct), they acknowledge the potential for a uniformity-tolerance dilemma where excessive uniformity might make the model intolerant to semantically similar samples.
- Future Work: The authors plan to further test language independence and explore multi-class multi-label SFT. They also suggest that task-specific approaches can be combined with their methods for further performance gains on individual tasks.
Summary updated