NADA: A Robust System for Non-referential Pronoun Detection

Shane Bergsma, David YarowskyDAARC 2011 · DOI

TL;DR

This paper introduces NADA, a novel and highly accurate system for detecting non-referential "it" pronouns in English text. By cleverly compressing massive web-scale N-gram features, NADA achieves state-of-the-art performance while remaining a fast, standalone system that doesn't require complex linguistic input.

What Problem Does This Paper Address?

In English, the pronoun "it" can be tricky. Sometimes it refers to a specific noun phrase (like "The dog wagged its tail"), but other times it's a "pleonastic" or "expletive" pronoun that doesn't refer to anything concrete (like in "It is important to maintain a stable price."). For tasks like information extraction, it's crucial to distinguish these two cases. If "it" refers to a noun phrase, we want to find that antecedent. If "it" is pleonastic, trying to find an antecedent is a waste of resources and can lead to errors. Existing coreference resolution systems often struggle with this, achieving only marginal improvements over simply guessing the most frequent class.

Methods

NADA combines two main types of features for its classification:

  • Lexical Features: These are binary indicators that capture the presence or absence of specific words or patterns within the entire sentence surrounding the pronoun "it." The authors found that using the whole sentence, rather than just a narrow window, significantly improved accuracy, mirroring human performance. They also experimented with various feature sets, ultimately omitting part-of-speech tags to allow the system to work on raw text.
  • Web Count Features: This is where NADA innovates. It leverages massive N-gram data from the web (like the Google N-gram Corpus) to learn patterns. The core idea is that for a given context (e.g., "it is ___ to"), certain words are much more likely to fill the blank depending on whether "it" is referential or non-referential. For instance, "able" might fill the blank in a referential context, while "important" might appear in a non-referential one. The challenge is that this web data is enormous. NADA's key contribution here is a sophisticated compression technique that reduces the 93GB raw N-gram data down to a mere 44MB file, allowing it to be loaded into memory and used efficiently. This compression involves lossy steps like keeping only 4-grams and filtering for specific keywords, as well as lossless steps like token encoding and value quantization.

NADA uses a regularized logistic regression model as its supervised classifier, trained on labeled data.

Datasets

  • Unlabeled Data: The Google N-gram Corpus (specifically, the 5-gram version) was used for extracting web count features.
  • Labeled Data:
    • BBN Pronoun Coreference Corpus: This was the primary dataset for training and evaluation. It contains antecedents for referential pronouns in the Wall Street Journal (WSJ) portion of the Penn Treebank. Instances of "it" were marked as referential if an antecedent was found, and non-referential otherwise. The corpus was split into training (3195 instances), development (1000 instances), and testing (3000 instances) sets.
  • ItBank Corpus: This corpus was used for out-of-domain evaluation. It includes "it" instances from news articles and the Slate portion of the American National Corpus. A subset of WSJ data from ItBank was also used as an in-domain test set.

Key Insights and Ideas

  • The Power of Compression: The most striking insight is how effectively massive web N-gram data can be compressed and utilized. The ability to shrink terabytes of data into tens of megabytes for practical use is a significant engineering feat.
  • Lexical vs. Count Features: While lexical features are important, web count features show a stronger ability to generalize to new domains, especially when combined with lexical features. This highlights the value of broad, unsupervised data for disambiguation tasks.
  • Sentence-Level Context is Key: For lexical features, using the entire sentence as context proved to be far more effective than relying on smaller windows, suggesting that the broader discourse context within a sentence is crucial for resolving the ambiguity of "it."
  • Integrated Approach Wins: Combining lexical and web count features yields the best performance, demonstrating that these two dominant paradigms for this task are complementary.

Key Results

  • Accuracy: NADA achieves state-of-the-art accuracy, outperforming previous systems by a significant margin. On the BBN dataset, NADA achieved 86.0% accuracy, a substantial improvement over the Charniak-Elsner system (73.4%) and the rule-based Paice-Husk system (75.2%). It also performed robustly on out-of-domain data.
  • Speed: NADA is remarkably fast, processing up to 20,000 sentences per second. This speed is achieved through efficient data compression and a streamlined architecture.
  • Robustness: The system performs well on different domains (news, Europarl) and doesn't require pre-parsed or tagged input, making it highly practical.

Limitations and Future Work

  • Limitations: The paper notes that the compression steps, particularly the lossy ones, might lead to a slight degradation in accuracy compared to using the full, uncompressed web data. However, they deem this a reasonable trade-off for practicality. The system is currently focused on the English pronoun "it."
  • Future Work: The authors hope that NADA will be adopted by other researchers and integrated into larger coreference resolution systems. The success of this approach suggests potential for extending it to other languages or other ambiguous pronouns. The efficient compression techniques could also be applied to other NLP tasks that rely on large N-gram data.

Summary updated

BibTeX
@inproceedings{bergsma2011nada,
  author = {Shane Bergsma and David Yarowsky},
  title = {{NADA: A Robust System for Non-referential Pronoun Detection}},
  booktitle = {DAARC},
  year = {2011},
  doi = {10.1007/978-3-642-25917-3\_2},
}
Figure