What is it? Disambiguating the different readings of the pronoun 'it'

Sharid Loáiciga, Liane Guillou, Christian HardmeierEMNLP 2017 · DOI

Introduction

In English, the pronoun "it" is a linguistic chameleon. It can be anaphoric, referring to a specific noun phrase (e.g., "I have a bicycle. It is red"); pleonastic, acting as a "dummy" subject with no semantic content (e.g., "It is raining"); or an event reference, pointing to a whole clause or action (e.g., "He lost his job. It came as a surprise").

While humans disambiguate these functions effortlessly, machines struggle. Most coreference resolution systems either ignore "it" or conflate event references and pleonastic uses as "non-referential." This ambiguity is a significant hurdle for Machine Translation; for example, translating "it" into gendered languages like German or French requires knowing its specific function to select the correct pronoun or grammatical structure.

Methodology

Researchers compared two distinct approaches to classify "it" into these three categories:

  1. MAXENT Classifier: A Maximum Entropy model utilizing 13 hand-engineered features, including token context, syntactic structure (e.g., presence of "that" complements), and semantic context derived from VerbNet to identify verbs that typically take abstract subjects.
  2. RNN (Bidirectional GRU): A deep learning model that processes context words to build internal representations. To overcome the scarcity of high-quality labeled data, the authors employed a "Silver-Standard" trick: they used the MAXENT model to label over 1 million sentences from the WMT16 corpus. The RNN was then trained on this "silver-standard" data (RNN-SILVER) and a combination of gold and silver data (RNN-COMBINED), a form of self-training designed to expose the network to more varied contexts.

Data

The study utilized two primary data sources:

  • Gold Standard: The ParCor corpus and DisCOMT2015.test dataset, providing approximately 2,500 manually annotated instances.
  • Silver Standard: A massive collection of 1.1 million sentences from WMT16 (Europarl, News, and TED talks), automatically labeled by the MAXENT classifier to provide scale for neural training.

Key Results and Model Performance

The evaluation revealed that model performance varies significantly depending on the target class. The MAXENT model excelled in identifying the majority class (anaphoric), while the RNN-COMBINED model demonstrated superior performance in the more challenging "event" category.

Comparison of model performance on development and test sets across different categories.

As shown in the performance table above, the MAXENT model achieved a peak F1-score of 0.735 for anaphoric "it" on the test set. However, the RNN-COMBINED model marked a breakthrough in event reference identification, reaching an F1 of 0.613, compared to MAXENT’s 0.542.

Analysis of Reference Relationships

A deeper analysis of specific reference types highlights the complementary strengths of the two models. While MAXENT is a specialist in nominal anaphora—identifying physical noun antecedents mentioned in the immediate context—the RNN-COMBINED model is significantly more robust at identifying clausal or sentential antecedents.

Performance breakdown by reference relationship, comparing MAXENT and RNN-COMBINED.

Specifically, the RNN-COMBINED model correctly identified 75% of "it" instances referring to sentential or clausal antecedents, whereas the MAXENT model only identified 59.1%. Conversely, MAXENT remained superior for NP antecedents in the previous two sentences (77% vs 54.8%).

Discussion and Future Work

The results suggest that "eventness" is difficult to capture with predefined linguistic categories, as evidenced by the fact that adding semantic features actually degraded MAXENT's performance on event references. Furthermore, the RNN's performance jump when trained on silver-standard data underscores the importance of a wide context window for pronoun disambiguation.

The primary limitation remains the reliance on the MAXENT model's potentially flawed predictions for silver-standard training. Future work aims to explore "co-training" strategies to refine the training sets and ultimately integrate this disambiguation system into full machine translation pipelines to improve pronoun accuracy in target languages.

Summary updated

BibTeX
@inproceedings{loaiciga2017what,
  author = {Sharid Loáiciga and Liane Guillou and Christian Hardmeier},
  title = {{What is it? Disambiguating the different readings of the pronoun 'it'}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1137},
}
Figure