Exploiting Named Entity Classes in CCG Surface Realization

Rajakrishnan Rajkumar, Michael White, Dominic EspinosaNAACL 2009

TL;DR

This paper shows how to significantly improve the quality of generating natural language sentences from logical forms using Combinatory Categorial Grammar (CCG). By cleverly incorporating information about named entities (like people, organizations, and dates) and their classes into language models and the "hypertagger" component of the system, they achieve state-of-the-art results and demonstrate a noticeable improvement in sentence fluency.

What Problem Does This Paper Address?

Generating natural language from structured representations (like logical forms) is a challenging task. A key difficulty lies in accurately realizing named entities (NEs) – words or phrases that refer to specific real-world things. Previous work has shown that better NE handling can improve generation quality, but the extent to which these improvements translate to human-perceived quality, especially in CCG, and how to best leverage NE classes (not just the entities themselves) remained open questions. The authors also highlight a general concern in the field: high BLEU scores don't always guarantee better-sounding output, making manual evaluation crucial.

Methods

The core of the paper revolves around enhancing the OpenCCG surface realization system. Their approach involves several key strategies:

  • Collapsing Named Entities: They selectively collapse certain multi-word NEs into single tokens. This is done carefully, only for NEs whose exact form is likely to be specified in the input, to avoid oversimplification. For example, "Texas Instruments Japan Ltd." might be collapsed, but "more than $10,000" would not be, as the "more than" part is compositional.
  • Class-Replaced Language Models: They create language models where NEs are replaced by their semantic class labels (e.g., "PERSON," "DATE:AGE"). This helps address data sparsity issues for rare NEs by grouping them with similar entities. These models are then interpolated with standard word-based and POS/supertag-based language models.
  • Class Features in Hypertagging: The "hypertagger" (which predicts lexical categories for words) is augmented with features derived from the semantic classes of NEs and their neighboring nodes in the logical form graph.

Datasets

  • CCGbank: Used for training and evaluation. The authors mention transferring BBN corpus NE annotations to the CCGbank.
  • BBN Corpus: Used for its NE annotations, which are then adapted for use with CCGbank.

Key Insights and Ideas

  • NE Classes Matter: Simply collapsing NEs is good, but using their classes in language models and the hypertagger provides a more substantial quality boost. This suggests that the semantic category of an entity is a strong signal for generation.
  • Targeted Collapsing is Key: Not all multi-word NEs should be collapsed. A selective approach, focusing on those likely to be atomic in the input, yields better results than collapsing all NEs.
  • Fluency Improvement is Real: The paper strongly advocates for manual evaluation alongside BLEU scores. Their manual evaluation confirmed that the BLEU score improvements directly translated to a significant increase in sentence fluency, addressing a common concern about BLEU's reliability.
  • Hypertagger Benefits from Class Info: While the hypertagger's per-predication accuracy might slightly decrease with collapsed NEs, the overall system benefits from incorporating class information as features.

Key Results

  • State-of-the-Art BLEU: The best system achieved a BLEU score of 0.8173 on Section 23 of the CCGbank, which was a significant improvement over previous results.
  • Language Model Impact: Class-replaced n-gram models provided a substantial boost, particularly on the collapsed corpus, outperforming simple 4-gram additions.
  • Hypertagger Impact: Adding class information to the hypertagger yielded an additional half BLEU point increase.
  • Manual Evaluation: Human judges rated the best system's output with mean scores of 4.4 for adequacy and 3.61 for fluency, compared to the baseline's 4.35 for adequacy and 3.36 for fluency. The increase in fluency was statistically significant (p = 0.015).
  • NE Error Reduction: The number of NE errors in the output was reduced from 238 to 99 for the best condition on the collapsed corpus, explaining much of the BLEU score improvement.

Limitations and Future Work

  • Hypertagger Performance Drop: The authors noted a slight decrease in hypertagger accuracy on the NE-collapsed corpus. They conjecture this might be resolved by incorporating corrections to CCGbank's NP structures, as suggested by Vadas and Curran (2008).
  • Future Work: Investigating the unexpected drop in hypertagger performance is a primary future direction. They also plan to continue exploring how to best leverage NE information for improved surface realization.

Summary updated

BibTeX
@inproceedings{rajkumar2009exploiting,
  author = {Rajakrishnan Rajkumar and Michael White and Dominic Espinosa},
  title = {{Exploiting Named Entity Classes in CCG Surface Realization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}
Figure