Abstract
Informal communication (e-mail, bulletin boards) poses a difficult learning environment because traditional grammatical and lexical information are noisy. Other information is necessary for tasks such as named entity detection. How topic-centric, or informative, a word is can be valuable information. It is well known that informative words are best modeled by "heavy-tailed" distributions, such as mixture models. However, informativeness scores do not take full advantage of this fact. We introduce a new informativeness score that directly utilizes mixture model likelihood to identify informative words. We use the task of extracting restaurant names from bulletin board posts as a way to determine effectiveness. We find that our "mixture score" is weakly effective alone and highly effective when combined with Inverse Document Frequency. We compare against other informativeness criteria and find that only Residual IDF is competitive against our combined IDF/Mixture score.
Introduction
Traditional Named Entity Recognition (NER) systems rely heavily on "surface features" like capitalization, punctuation, and formal grammar. While these cues are reliable in structured text like news articles, they often fail in informal environments such as bulletin boards, emails, or newsgroups. In these "noisy" settings, users frequently ignore standard conventions, making it difficult to identify entities like restaurant names using traditional methods.
Researchers from MIT’s CSAIL proposed a shift toward statistical measures of informativeness—identifying how "topic-centric" a word is based on its distribution across a corpus. The goal is to find "bursty" words that appear frequently in specific contexts but are absent elsewhere, regardless of their capitalization.
Methodology: The Mixture Score
The core innovation of this work is the Mixture Score. This metric is based on the intuition that informative words follow a "heavy-tailed" distribution. While most words can be modeled by a simple unigram distribution (constant probability), informative words operate in two modes: a "high frequency" mode in relevant documents and a "low frequency" mode in irrelevant ones.
To capture this, the authors compare two models:
- Unigram Model: Assumes a single, constant probability for a word.
- Mixture Model: Uses two unigram components (relevant vs. irrelevant) and a mixing parameter.
The Mixture Score is the log-odds ratio of the likelihood of the data under the mixture model versus the simple unigram model. A high score indicates that the word is highly informative and topic-specific.
Experimental Setup
The researchers tested their approach on the task of extracting restaurant names from a moderated bulletin board. The dataset consisted of 615 posts (approximately 56,000 tokens), with 1,968 tokens manually labeled as restaurant names. Many of these names were common words (e.g., "Fish," "Speed") that would easily be missed by dictionary-based or capitalization-reliant systems.
Results and Discussion
A key finding was the relationship between the Mixture Score and Inverse Document Frequency (IDF). While IDF measures rarity, the Mixture Score measures "burstiness." Surprisingly, these two metrics were found to be almost entirely independent, with a correlation coefficient of only -0.0139.
Because they capture different aspects of term importance, combining them ($IDF \times Mixture$) created a powerful hybrid feature. When integrated into a Regularized Least Squares Classifier (RLSC), this combined feature improved the F1-score from a baseline of 55.04% to 59.30%.
As shown in the rankings above, the Mixture Score is exceptionally effective at identifying restaurant-related tokens. In the top 10 words ranked by the score, 7 were actual components of restaurant names (e.g., Sichuan, Villa, Tokyo). The only other metric that remained competitive was Residual IDF, which also models the deviation of a word's distribution from a random (Poisson) model.
Conclusion
This research demonstrates that statistical informativeness is a vital feature for NER in noisy text environments. By modeling the "burstiness" of terms through a mixture model, the Mixture Score provides a signal that is complementary to IDF. Future work may involve applying these features to more sophisticated sequence models like CRFs to further improve absolute performance in informal text extraction tasks.
Summary updated