Toward the Essential Nature of Statistical Knowledge in Sense Resolution

Jill Fain LehmanAAAI 1994

In the early 1990s, the field of Natural Language Processing (NLP) faced a significant divide. On one side were knowledge-rich symbolic systems, which offered high accuracy but struggled to scale. On the other were knowledge-lean statistical systems, which scaled efficiently but were often viewed as lacking true linguistic "understanding." Curiously, despite their mathematical differences, various statistical methods—Bayesian classifiers, neural networks, and content vectors—all seemed to converge at a similar performance ceiling.

This paper investigates the "essential nature" of the knowledge these statistical techniques utilize. Jill Fain Lehman argues that the perceived power of complex statistical models for word sense disambiguation (WSD) does not stem from sophisticated probability theory, but rather from a simple form of episodic memory: co-occurrence counts.

Stripping Down Statistical Knowledge

To isolate the core drivers of performance, Lehman introduced two minimalist models designed to test the limits of simple memory:

  1. EM0 (Existence Only): This model records only whether a word has ever appeared with a specific sense in the training data. To resolve a sense in a new sentence, it counts how many words in that sentence have a recorded "existence" for each possible sense and selects the sense with the highest count.
  2. EM1 (Frequency Only): A slightly more advanced version that tracks the raw frequency of word-sense pairs. Unlike standard classifiers, it lacks meta-information such as total corpus size or probability distributions.
Recall performance across training set sizes for various models.

As shown in the figure above, the performance of EM1 closely tracks that of much more complex models like Bayesian classifiers and Neural Networks, even as the training set size increases. This suggests that raw frequency is the primary contributor to the success of statistical WSD.

Experimental Benchmarks

The study utilized two primary benchmarks from the Wall Street Journal (WSJ) and APHB corpora:

  • The "duty" corpus: A pilot dataset of 140 instances categorized into three senses (tax, responsibility, and work).
  • The "line" corpus: A larger dataset of 4,136 instances covering six senses of the word "line" (e.g., cord, division, phone, product).

Key Insights: The Power of Presence

The most striking finding is that simply knowing a word can co-occur with a sense (EM0) provides a massive portion of the total resolution power. However, EM0’s performance degrades as polysemy (the number of possible senses) increases. This is not due to incorrect knowledge, but rather a "tie" problem: as more senses are introduced, EM0 is more likely to find an equal number of co-occurring words for multiple senses, forcing a random guess.

Comparative performance of statistical and frequency-based models across varying levels of polysemy.

Results and the "Tie" Problem

In the 6-sense "line" corpus experiment, complex models achieved 71-76% recall, while EM1 achieved 70%. EM0 lagged behind at 62%. However, Lehman introduced the Effective Recall Rate (ERR) to measure accuracy only when a model makes an unequivocal decision (no ties). When ties were removed, EM0’s accuracy jumped to 73%, nearly matching the heavyweights.

This proves that the "statistical" advantage of complex models is primarily a more robust way to break ties rather than a fundamentally different kind of knowledge. The "existence" of a word-sense pair in memory is the primary driver of success.

Conclusion

The research suggests that for cognitive models, the goal should be to move from a "corpus of sentences" to a "corpus of experiences." By recognizing that simple episodic memory provides the bulk of statistical power, researchers can integrate these "knowledge-lean" benefits into symbolic systems without the overhead of complex probability distributions. Future work remains in managing polysemy to ensure that simple existence-based memory remains viable in increasingly complex linguistic contexts.

Summary updated

BibTeX
@inproceedings{lehman1994toward,
  author = {Jill Fain Lehman},
  title = {{Toward the Essential Nature of Statistical Knowledge in Sense Resolution}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1994},
}
Figure