TL;DR
This paper tackles the challenge of selecting the most appropriate linguistic form (e.g., pronoun, proper name, definite description) for referring to entities in text. By systematically evaluating existing features used in machine learning models for this task, the authors propose a concise "consensus" feature set that achieves comparable performance to larger, more complex sets, offering insights for both computational linguistics and natural language generation research.
What Problem Does This Paper Address?
Referring Expression Generation (REG) is a crucial task in Natural Language Generation (NLG), aiming to produce natural-sounding language that refers to specific entities. A key sub-problem within REG is the "Selection of Referential Form" (SRF): deciding how to refer to an entity. For instance, should we say "he," "John," or "the man"? This choice is influenced by various linguistic factors, such as how prominent the entity is in the discourse. While many computational models exist for SRF, they often use diverse and sometimes opaque feature sets. This paper seeks to bridge the gap between linguistic theories of reference and computational approaches by systematically evaluating the importance of different features used in SRF models, ultimately aiming to identify a core set of linguistically interpretable features that drive performance.
Methods
The authors conducted a systematic evaluation of feature-based machine learning algorithms for SRF. They began by gathering feature sets from existing SRF studies, primarily from the GREC challenges and other relevant literature, resulting in 65 distinct linguistic features grouped into 9 broad categories. These features were then re-implemented and applied to the OntoNotes corpus.
To assess feature importance, two primary methods were employed:
- Rank Features by Importance (RFI): This method leverages the built-in permutation importance of Random Forests (RF). It works by building an RF model, computing its accuracy, then permuting the values of a specific feature to break its link with the outcome. The difference in accuracy before and after permutation indicates the feature's importance.
- Sequential Forward Search (SFS): This greedy algorithm starts with an empty feature set and iteratively adds the feature that provides the greatest improvement in accuracy, stopping when further additions yield minimal gains.
Random Forests were chosen as the classification algorithm due to their ability to reliably compute feature importance and their strong performance.
Datasets
The primary dataset used in this study is the Wall Street Journal portion of OntoNotes. This corpus was chosen for its rich annotations, including structural information (syntax, predicate argument structure) and shallow semantics, which are valuable for extracting features. The authors also incorporated information from the PDTB parser to extract paragraph information.
The study did not introduce a new benchmark dataset but rather focused on evaluating existing feature sets on a well-established corpus.
Key Insights and Ideas
- Smaller Feature Sets Can Be Sufficient: A key finding is that models can achieve nearly identical performance with significantly smaller feature sets compared to the original, larger ones. This suggests that many features might be redundant or contribute little to the predictive power.
- Higher-Level Recency Features Dominate: Features related to "higher-level" units like sentences and paragraphs (e.g., sentence distance) are consistently ranked as more important than "lower-level" recency features (e.g., word distance). This aligns with linguistic intuitions about discourse structure.
- Grammatical Role of the Current Mention is Crucial: The grammatical role of the current mention being referred to is more important for predicting referential form than the grammatical role of its antecedent. This contrasts with some previous research that focused more on the antecedent's subjecthood.
- Animacy and Plurality Matter: Inherent features like animacy and plurality play significant roles. The importance of animacy is consistent with prior work showing pronouns are more frequent for animate referents. The role of plurality is a more surprising finding, potentially linked to how conjoined noun phrases are treated as a prominent group.
- Linguistic Interpretability is Key: The study emphasizes the value of linguistically interpretable features, aiming to connect computational models with linguistic theories of reference.
Key Results
The paper evaluated seven feature sets, resulting in 65 distinct features. After applying RFI and SFS, a "consensus" feature set of 6 features was proposed. This consensus set, derived from 4 broad linguistic categories (grammatical role, inherent features, antecedent form, and recency), achieved an accuracy of 0.776 when tested using SFS. This is comparable to the best-performing original model (Kibrik) which used 17 features and achieved an accuracy of 0.793. In some cases, subsets of features even outperformed the original full sets. For instance, the Kibrik model with only 6 consensus features achieved an accuracy of 0.776, very close to its original performance of 0.793.
Limitations and Future Work
- Exclusion of Deep Learning: The study explicitly excluded Deep Learning methods, stating that they currently offer less opportunity for linguistic interpretation. This means the findings are specific to feature-based ML models.
- Feature Selection Subjectivity: While the paper aims for systematic evaluation, the initial selection of feature sets was based on criteria like SRF being the main objective and the use of interpretable features.
- Future Work: The authors suggest that computational and theoretical studies of language should continue to inspire each other. The proposed consensus feature set has implications for both feature-based SRF research and linguistics, potentially guiding future model development and theoretical investigations into reference. The paper also hints at the possibility of further refining feature selection to achieve an optimal compromise between feature count and performance.
Summary updated