TL;DR
This paper investigates how to explicitly incorporate entity information into neural language models (LMs). The authors find that while current entity-aware LMs don't outperform standard LMs in overall perplexity, a simplified approach (SetLM) shows promise in accurately identifying entity mentions, suggesting a more implicit handling of entity decisions might be beneficial.
What Problem Does This Paper Address?
Human language is deeply intertwined with our understanding of entities in the world and how they relate to each other (coreference). While traditional coreference resolution has been a long-standing research area, its integration into language modeling (LM) is a more recent development. The challenge lies in effectively teaching LMs to recognize and reason about entities within a text, going beyond simply predicting the next word based on its immediate context. Existing approaches often introduce latent variables to decide if a token is part of an entity and which entity it refers to. However, the effectiveness and practical benefits of these explicit entity modeling techniques in general language modeling tasks remain unclear. This paper aims to systematically compare different approaches to explicit entity modeling within LMs and analyze their performance, particularly focusing on the challenges of entity detection and coreference.
Methods
The authors tackle this problem by:
- Replicating Existing Models: They independently reimplement two prominent entity-aware LMs: YangLM (Yang et al., 2016) and EntityNLM (Ji et al., 2017). This ensures a fair comparison under a controlled experimental setup.
- Introducing a Simplified Variant (SetLM): They propose SetLM, a novel architecture that simplifies the entity decision process. Instead of a latent variable to decide if a token is an entity, SetLM introduces a learned embedding for non-entity tokens. This aims to reduce the complexity of the decision-making process.
- Comparative Evaluation: All three entity-aware models are compared against each other and against standard RNN-LMs (with varying hidden sizes) using perplexity as the primary metric. They also perform a detailed analysis of individual decision steps, including entity yes/no prediction, entity choice accuracy, and mention length prediction.
Datasets
The models were trained, optimized, and evaluated on the English subset of the OntoNotes 5.0 corpus. This dataset is notable for containing anaphoric coreference annotations within documents. The standard CoNLL-2012 split was used for train, development, and test sets. Preprocessing involved lowercasing, replacing numbers with a special symbol, and replacing rare words with a special token, resulting in a vocabulary of 11,539. Mention length was capped at 25 tokens, with nested mentions being retained.
Key Insights and Ideas
- Entity Detection is Hard: The decision of whether a token belongs to an entity mention is a significant bottleneck, leading to low precision and recall. This suggests that current models struggle with this fundamental step, causing cascading errors.
- Implicit Handling Might Be Better: SetLM, which simplifies the entity decision by using a learned embedding for non-entity tokens rather than a discrete latent variable, achieves the best F1 score for the entity yes/no prediction. This hints that a more implicit approach to entity detection might be more robust.
- Standard LMs are Competitive: Even with explicit entity modeling, none of the proposed models outperform a standard RNN-LM with a sufficiently large number of hidden units. This raises questions about the immediate benefits of explicit entity integration for general language modeling tasks.
- Gold Information Boosts Performance: The re-implemented EntityNLM and YangLM models show improved performance when given access to gold entity lists during evaluation. This suggests that their reported improvements might be partly due to this external information, rather than solely their architectural design.
Key Results
- Perplexity: The optimized baseline RNN-LM (RNN-512) achieved the lowest perplexity (96), significantly outperforming all entity-aware models. Among the entity-aware models, YangLM (own) performed best with a perplexity of 107, followed by SetLM (114) and EntityNLM (own) (131).
- Entity Yes/No Prediction: SetLM achieved the highest F1 score (48.1%) for the entity yes/no prediction task, indicating it handles this crucial decision most effectively among the entity-aware models. YangLM (own) and EntityNLM (own) had F1 scores of 40.3% and 45.3%, respectively.
- Entity Choice Accuracy: The accuracy for choosing the correct entity from a set was relatively high across models, with EntityNLM (own) achieving the highest accuracy (67.8%).
- Length Prediction: EntityNLM's length prediction was correct in 59.4% of cases, with an average error of 2.85 tokens from the gold length.
Limitations and Future Work
- Limited Gains from Explicit Modeling: The authors conclude that explicit entity modeling does not currently improve general language modeling performance (perplexity) compared to well-tuned standard RNN-LMs.
- Error Propagation: The significant errors in entity detection can lead to substantial error propagation, outweighing the potential benefits of explicit entity information, especially in shorter texts.
- Hyperparameter Sensitivity: Finding optimal hyperparameter settings for entity-aware models proved challenging, with oscillations in scores and a lack of significant improvement from increasing hidden units without pre-training.
- Future Directions: The authors suggest that explicit entity representations might be more beneficial in longer texts or for tasks involving complex interactions between entities over long distances. They also highlight the need for models that are less prone to error propagation and for evaluations on larger, higher-quality annotated datasets and different genres. The potential for continuous entity representations to capture long-range dependencies is also noted as a promising avenue.
Summary updated