TL;DR
This paper proposes a method to improve natural language understanding models' performance on rare or unseen words by incorporating their definitions. By augmenting training data with word definitions, the model can better handle examples that would otherwise be misclassified due to poor word embeddings, recovering most of the performance lost from completely scrambled words.
What Problem Does This Paper Address?
Modern natural language understanding (NLU) models heavily rely on pre-trained word embeddings. These embeddings capture semantic relationships between words based on their usage in massive text corpora. However, this reliance becomes a bottleneck when NLU models encounter words that are rare, technical, or simply not present in the pre-training data (e.g., "COVID" before 2020). Such models struggle to understand the meaning of these words, leading to errors in tasks like natural language inference (NLI) or sentiment classification. The paper aims to bridge this gap by enabling models to leverage external knowledge, specifically word definitions, to compensate for these weak or absent embeddings.
Methods
The core of the proposed method involves creating an "enhanced training set" (T') from an original training set (T). For each example in T, the authors identify "critical words" – words that are crucial for classification and appear only once. They use a technique inspired by input marginalization to identify these words: they check if replacing a word with another likely word changes the classification outcome.
Once critical words are identified, their definitions are retrieved (from Simple English Wiktionary or English Wiktionary). The original word embeddings are then effectively "obfuscated" (by scrambling them in experiments) to force the model to rely on the provided definitions. The enhanced training set T' consists of pairs of (original sequence, definition) and (sequence with critical word replaced, definition of replaced word). This allows the model to learn to associate definitions with their corresponding words and use them for classification. The authors also explore a protocol where both the critical words and their definitions are scrambled to prevent the model from relying on any pre-existing embeddings.
Datasets
- SNLI (Stanford Natural Language Inference): Used for training and evaluating the NLI models.
- SNLItrue_crit: A subset of SNLI used for testing, constrained to include only examples where the original label is preserved.
- SNLInew_crit: A subset of SNLI used to test if definitions are simply being memorized, by selecting examples where the defined word was not seen in training definitions.
- SNLItrue_multi: A subset of SNLI designed to test performance on words whose lemma form spans multiple subwords, hypothesizing these are less well-understood by language models.
Key Insights and Ideas
- Definitions as a powerful fallback: Word definitions can significantly compensate for poor or absent word embeddings, allowing models to perform tasks they would otherwise fail at.
- Forcing definition reliance: Scrambling word embeddings is crucial to ensure that the model actually learns to use the provided definitions, rather than relying on its pre-trained knowledge.
- Definition reasoning vs. simple substitution: Understanding a text definition is harder for models than simply substituting a word with its synonym or a related concept.
- Memorization is not the whole story: While models can "remember" definitions from training, they also demonstrate an ability to reason with them, as shown by performance on the
SNLInew_critsubset.
Key Results
- When words are scrambled and no definitions are provided, NLI accuracy drops significantly (e.g., from 85.1% to 36.9%).
- The proposed method, using definitions with scrambled words, recovers most of this lost performance, achieving 81.2% accuracy on the
SNLItrue_crittest set. - Training with definitions but without scrambling words yields slightly higher accuracy (85.2%) but the model fails to use definitions when words are scrambled (51.4% accuracy).
- The method is effective even on a "hard subset" of SNLI (
SNLItrue_multi) designed to test challenging word forms, though performance is still lower than with normal embeddings.
Limitations and Future Work
- Integration with existing embeddings: The paper notes that they have not discovered a way to combine the benefit of definitions with the knowledge in original word embeddings. This suggests a direction for future work.
- Word selection: The current method identifies critical words automatically. For practical applications, it would be beneficial to select words that would most benefit from definitions.
- Multiple definitions: The current approach handles one definition per word. Applications might require models to accept and process multiple definitions.
- Specialized training: The paper highlights that definitions are not well utilized without special training. Developing models that can leverage definitions more naturally, without explicit training protocols, is a key area for future research.
Summary updated