TL;DR
This paper introduces "reversal-based" data augmentation for Natural Language Inference (NLI) tasks, specifically focusing on problems with "opposable" classes (like "positive" vs. "negative"). They propose a novel method, "main verb inversion," that effectively negates sentences to create examples for the opposing class, significantly boosting classifier performance, especially on imbalanced datasets.
What Problem Does This Paper Address?
In Natural Language Processing (NLP), high-performing models often require large amounts of training data. While data augmentation is common in fields like computer vision, it's less established in NLP. Existing NLP augmentation methods primarily focus on "rephrasing" sentences to create variations that preserve the original meaning. This is problematic for tasks with opposable classes, as these methods can only generate more examples of the same class, leaving the opposing class under-represented. The authors tackle this by proposing methods to reverse the meaning of sentences, thereby generating valuable training data for the under-represented, opposing classes.
Methods
The paper proposes two main reversal-based augmentation techniques, building upon existing work:
- Main Verb Inversion (Enhancement of "Add Negation"): This method focuses on negating the main verb of a sentence. If a sentence lacks negation, it's added. Conversely, if negation is present, it's removed. This is a more targeted approach than simply adding negation, aiming to create grammatically sound, meaningful inversions.
- Adjective and Adverb Antonymy (Enhancement of "Antonym"): This approach aims to improve upon existing antonym-based methods, which often produce grammatically incorrect or nonsensical sentences. The authors refine this by carefully selecting antonyms, preserving comparative/superlative forms of adjectives, and ensuring the antonym is directly connected to the sentence's root. They also acknowledge that previous antonym methods sometimes picked inappropriate words due to polysemy or left-to-right processing.
Datasets
The authors collected a new dataset specifically for this research on the Amazon Mechanical Turk platform. This dataset consists of 851 sentences describing logical expressions, categorized into six classes that form three "naturally opposable" pairs (e.g., "Equal" vs. "Not Equal"). This dataset is publicly available.
Key Insights and Ideas
- The Value of "Opposable" Classes: For tasks with inherently opposing classes, simply rephrasing sentences isn't enough. Generating examples for the opposite class is crucial for robust model training.
- "Main Verb Inversion" is Promising: This method proved to be a significant improvement over existing negation strategies and consistently boosted performance.
- Antonymy is Tricky: While the idea of using antonyms is intuitive, previous implementations often resulted in low-quality, nonsensical sentences, hindering their effectiveness. The authors' refined approach shows promise but is less impactful than verb inversion.
- Augmentation for Imbalanced Data: Reversal-based augmentation, particularly main verb inversion, is especially beneficial when dealing with imbalanced datasets, where one class has significantly fewer examples than another.
Key Results
- Experiment #1 (Reversal vs. No Augmentation):
- The original "Add negation" and "Antonym" methods by Niu and Bansal (2018) showed unreliable performance, often worsening results.
- "Main verb inversion" consistently improved performance across various undersampling scenarios, achieving F1 scores significantly better than the baseline. For example, with 100% data, it reached an F1 of 63.57, compared to the baseline's 56.05.
- "Adjective and adverb antonymy" showed minimal improvement and was dropped from further experiments.
- Experiment #2 (Reversal + Rephrasing):
- Combining "Main verb inversion" with EDA (a popular rephrasing-based approach) resulted in a 3-11% improvement over EDA alone.
- "Add negation" combined with EDA did not show significant improvement.
Limitations and Future Work
- Domain Specificity: The authors acknowledge that "main verb inversion" might be difficult to apply to less technical language (e.g., tweets or movie reviews) in its current form.
- Antonymy Challenges: While their refined antonymy approach is better, it still faces challenges, and its effectiveness is less pronounced than verb inversion.
- Future Directions: The authors suggest exploring ways to extend "main verb inversion" to other application areas and less technical language. They also propose investigating the reasons for the ineffectiveness of antonym-based approaches in more detail. An interesting future direction is to explore black-box models (like sequence-to-sequence models) instead of heuristic rules for generating inverted sentences.
Summary updated