Abstract
Word Sense Disambiguation (WSD) is a critical task in natural language understanding, often considered "solved" by modern contextualized embeddings. However, existing benchmarks may rely on simple statistical cues rather than true contextual comprehension. This paper introduces the FOOL (FOur-fold Obscure Lexical) dataset, which challenges models with adversarial contexts. Our evaluation of various architectures—including encoders like BERT and decoders like GPT-4o—reveals that while massive models lead in complex reasoning, smaller encoder models are often more robust against local linguistic distractors.
1. Introduction
Word Sense Disambiguation (WSD) involves determining the correct meaning of a polysemous word based on its surrounding context. While performance on standard benchmarks is high, these datasets often lack the complexity to distinguish between genuine semantic understanding and simple pattern matching. For instance, a model might correctly identify "apple" as a fruit simply because the word "crisp" appears nearby. This study investigates whether models are robust enough to ignore such "adversarial" decoys or if they are merely being "fooled" by local word associations.
2. The FOOL Dataset
The core contribution of this work is the FOOL dataset, focusing on 20 homonyms with two distinct, unrelated meanings. The dataset is structured into four levels of increasing difficulty to probe model robustness:
- Set 1 (Baseline): Standard sentences where context clearly indicates the sense.
- Set 2 (Aligned): Sentences enhanced with helpful adjectives (e.g., "crisp apple").
- Set 3 (Artificial Adversarial): Sentences containing misleading adjectives from the opposing sense (e.g., "innovative apple" referring to the fruit).
- Set 4 (Realistic Adversarial): Human-crafted sentences with complex, conflicting elements of both senses.
3. Methodology
We evaluated two primary architectural paradigms:
- Encoder Models (Probing): Models such as BERT, RoBERTa, and the T5 family were evaluated by extracting contextualized embeddings of target words and applying a k-Nearest Neighbor (kNN) classifier.
- Decoder Models (Prompting): Large Language Models (LLMs) including GPT-4o, Llama3, and Mixtral were tested via zero-shot prompting, requiring the model to select the correct sense from provided options.
4. Results and Analysis
Our findings highlight a significant "adversarial cliff" where performance drops as context becomes more misleading. While most models achieve near-perfect F1-scores on Set 1, performance degrades sharply on Sets 3 and 4.
4.1 The "Local Distractor" Trap
Models frequently over-rely on immediate neighbors. In Set 3, the introduction of a single misleading adjective caused substantial performance declines. Interestingly, encoder models like BERT and T5 demonstrated surprising resilience here. Because these models utilize bi-directional context to form representations, they are less likely to be swayed by a single local word compared to some larger decoder models.
4.2 Reasoning vs. Pattern Matching
GPT-4o emerged as the most robust model for Set 4 (Realistic Adversarial), maintaining an F1-score of 0.956. This suggests that while it can be tripped by simple local "tricks" in Set 3, its superior reasoning capabilities allow it to resolve more complex, human-like linguistic conflicts. Conversely, a "Small Model Victory" was observed on Set 3, where BERT-large (0.874) and T5-large (0.896) outperformed GPT-4o (0.860), indicating that specialized architectures can hold an edge in specific robustness tasks.
5. Conclusion
The FOOL dataset exposes vulnerabilities in current LMs, showing that scale does not automatically equate to linguistic robustness. While LLMs like GPT-4o lead in complex reasoning, smaller encoder models provide competitive or superior performance against local adversarial distractions. Future work will expand FOOL to include more diverse parts of speech and finer-grained sense distinctions.
Summary updated