Introduction
In the field of Natural Language Processing (NLP), there is a prevailing "folk wisdom" that neural models are inherently superior to pattern-based systems because of their superior recall. Yoav Goldberg challenges this notion, arguing that our current understanding of recall is too broad. He proposes a distinction between two types: d-recall (diversity) and e-recall (exhaustiveness).
The Two Recalls
Goldberg defines these terms to highlight a critical gap in model performance:
- d-recall (Diversity): The ability to identify many different linguistic variations of a single concept (e.g., recognizing that "X graduated from Y," "X is an alumnus of Y," and "X studied at Y" all describe the same relationship). Neural models excel here.
- e-recall (Exhaustiveness): The ability to reliably find every instance of a known pattern. While pattern-based systems are naturally exhaustive, neural models often exhibit a "lack of systematicity," missing simple instances of a relationship they have successfully identified in more complex contexts.
Methodology: Testing for Systematicity
To demonstrate this gap, Goldberg compared a neural-based approach (RoBERTa-large fine-tuned on SQuAD v2.0) against a pattern-based system (SPIKE) that uses syntactic dependency trees. The experiment involved using SPIKE to find thousands of sentences matching a specific linguistic structure (e.g., "Subject was educated at Object") and then testing whether the neural model could extract the same information. If the neural model returns "No-Answer" for a sentence the pattern-based system easily caught, it demonstrates a failure in e-recall.
Key Results and Neural Brittleness
The results revealed significant "blind spots" in neural models. In a test using 5,000 Wikipedia sentences matching the "educated at" pattern, the RoBERTa model failed to find an answer in approximately 12% of cases. Similar failures occurred in biomedical contexts (DISEASE treated with CHEMICAL), where the model missed over 700 instances of a relationship.
Even massive models like GPT-3 (specifically text-davinci-003) are not immune. In a subset of 100 clear "educated at" sentences, the model failed on 2% of cases by swapping arguments—extracting the school name when asked for the person's name. This highlights a fundamental brittleness where models fail on very direct, concise sentences despite succeeding on much more complex ones.
The Evaluation Gap
Why has this gone unnoticed? Goldberg argues that current evaluation benchmarks, such as SQuAD, are heavily biased toward d-recall. Human annotators are typically encouraged to vary their phrasing, leading to datasets that reward diversity but fail to penalize a lack of exhaustiveness. We aren't testing if models are exhaustive because our tests don't contain enough repetitive, "boring" examples. Consequently, we are building systems that lack the reliability of simpler, pattern-based methods.
Conclusion
Goldberg's position paper serves as a call to action for the NLP community. To build truly robust systems, researchers must adopt the vocabulary of d-recall and e-recall and develop new benchmarks specifically designed to measure exhaustiveness and systematicity across tasks like Named Entity Recognition (NER) and coreference resolution. The goal is to ensure neural models don't lose the reliability of pattern-based systems while retaining their powerful generalization abilities.
Summary updated