Abstract
We present a novel named entity recognition (NER) system, and its participation in the Emerging and Rare Entity Recognition shared task, hosted at the 2017 EMNLP Workshop on Noisy User Generated Text (W-NUT). With a specialized evaluation highlighting performance on rare, and sparsely-occurring named entities, this task provided an excellent opportunity to build out a newly-developed statistical algorithm and benchmark it against the state-of-the-art. Powered by flexible context features of word forms, our system's capacity for identifying neverbefore-seen entities made it well suited for the task. Since the system was only developed to recognize a limited number of named entity types, its performance was lower overall. However, performance was competitive on the categories trained, indicating potential for future development.
Introduction
Standard Named Entity Recognition (NER) systems excel at identifying common entities in structured, formal text like news reports. However, their performance often degrades when applied to "noisy" user-generated content from social media or forums. In these domains, grammar is inconsistent and "emerging" entities—such as new viral personalities or niche products—appear frequently without prior representation in training data. These Out-of-Vocabulary (OOV) entities represent a significant challenge for traditional models that rely on fixed dictionaries or rigid word-level features.
This paper introduces a novel NER system designed specifically to address the "long tail" of rare entities. By focusing on "internal context"—the character-level patterns within words—the system can identify entities it has never encountered before by recognizing structural signals in their spelling.
Methodology
The proposed Context-Sensitive NER shifts the focus from the word as a whole to its sub-word components. This approach is built on several core components:
- Internal Context Models: Instead of relying solely on surrounding words (external context), the model analyzes character sequences within a token. For a token $w$ of length $n$, the system defines a set of $2^{n-1}$ contexts created by removing contiguous sub-sequences of characters. This allows the model to identify patterns like common suffixes (e.g., "-son" in names) that signal an entity type.
- Likelihood Estimation: The system calculates the probability of a specific entity tag (e.g., "B-person") given these internal character contexts. This enables "zero-shot" recognition for rare names that share structural similarities with known entities.
- Harmonic Mean for Multi-word Entities: To identify entities spanning multiple words (e.g., "New York City"), the system assesses the group using the harmonic mean of the component-word likelihoods.
- Conflict Resolution: A hierarchical logic is applied to resolve overlapping predictions, prioritizing the first appearing prediction, then the longest prediction, and finally the one with the highest likelihood.
Experimental Setup and Datasets
The model was evaluated as part of the W-NUT 2017 Shared Task. The dataset distribution across categories is detailed in Figure 1.
The system utilized a mix of gold-standard data and supplemental lexica:
- W-NUT 2017 Data: The primary benchmark featuring six entity types: Corporation, Creative Work, Group, Location, Person, and Product.
- Historical NER Sets: Data from W-NUT 2016, CoNLL 2003, WikiNER, and the Groningen Meaning Bank were used for broader training.
- Supplemental Lexica: Large-scale gazetteers were incorporated using a weighting scheme to balance these massive lists against gold-standard training samples.
Results and Discussion
The system, identified as Drexel-CCI in the shared task, demonstrated a specialized performance profile. While the overall F1 score (25.53) was lower than top-tier ensemble systems, the model was highly competitive in the specific categories for which it was optimized.
Key performance insights include:
- Niche Strength: On Location entities, the system achieved a precision of 58.21%, outperforming several other teams. On Person entities, it reached 49.58% precision.
- Optimization Sensitivity: The system recorded scores of 0 for "Corporation" and "Creative Work." The authors found that their optimization procedure had almost no predictive power for these types during development, likely because these categories rely more on external context (usage) than internal spelling patterns. Consequently, the model was restrained from making predictions in these categories to maintain precision elsewhere.
- Precision-Recall Trade-off: In noisy domains, the internal context approach allowed for high precision in specific categories, even if the overall recall was lower than state-of-the-art models that utilize more complex external features.
Conclusion and Future Work
The primary limitation of the current system was its narrow focus, as it was only fully optimized for two of the six entity types. Future development will focus on creating Hybrid Contexts that combine internal character-level features with external word-level features to capture both the spelling and the usage of a word. Additionally, applying the weighted gazetteer approach to more diverse categories and refining multi-word entity logic are expected to improve recall across the board.
Summary updated