Generating CCG Categories

Yufang Liu, Tao Ji, Yuanbin Wu, Man Lan

Abstract

Previous CCG supertaggers usually predict categories using multi-class classification. Despite their simplicity, internal structures of categories are usually ignored. The rich semantics inside these structures may help us to better handle relations among categories and bring more robustness into existing supertaggers. In this work, we propose to generate categories rather than classify them: each category is decomposed into a sequence of smaller atomic tags, and the tagger aims to generate the correct sequence. We show that with this finer view on categories, annotations of different categories could be shared and interactions with sentence contexts could be enhanced. The proposed category generator is able to achieve state-of-the-art tagging (95.5% accuracy) and parsing (89.8% labeled F1) performances on the standard CCGBank. Furthermore, its performances on infrequent (even unseen) categories, out-of-domain texts and low resource language give promising results on introducing generation models to the general CCG analyses.

Introduction

Combinatory Categorial Grammar (CCG) supertagging is a critical step in syntactic parsing, often referred to as "almost parsing." It involves assigning complex syntactic categories to words, which dictate how they combine with neighbors. Traditionally, this task is framed as a multi-class classification problem. However, this approach treats each category as an opaque, monolithic label.

Internal structures of CCG categories showing shared atomic components.

As shown in Figure 1, categories like (S\NP)/NP and (NP\NP)/NP share significant internal structure. A standard classifier sees these as unrelated, failing to exploit the shared "combining strategy"—taking an argument from the right, then the left. This leads to brittleness when encountering rare tags and an inability to handle categories not seen during training.

Methodology

To address these limitations, we propose shifting from classification to sequence generation. By decomposing complex tags into atomic building blocks (like (, S, \, NP, /), the model can learn the underlying grammar of the tags themselves. We explored two primary architectures:

  1. Tag-wise Generator: An LSTM-based decoder that generates atomic sequences for each word, utilizing an attention mechanism to maintain context from the input sentence.
  2. Transition-based Generator: A more constrained approach using a stack-based transition system to ensure the mathematical validity of generated CCG structures.

We also evaluated different "decoding oracles" to define the generation alphabet, ranging from simple atomic symbols (AC) to n-grams (NG) that capture common sub-sequences like NP\NP to improve efficiency.

Results and Discussion

Our generative approach achieved a state-of-the-art accuracy of 95.5% on the CCGBank benchmark, rising to 96.05% when integrated with BERT.

Accuracy across different category lengths, demonstrating robustness on complex tags.

One of the most significant advantages of this method is its performance on the "long tail" of syntactic structures. As illustrated in Figure 2, the generative models maintain higher accuracy as category length (and thus complexity) increases compared to traditional methods. Specifically, on rare categories appearing fewer than 100 times, the generator outperformed the classifier by over 5% (65.83% vs 60.41%). Most notably, the model demonstrated "zero-shot" capabilities, successfully constructing valid categories that were entirely absent from the training data.

Conclusion

By treating CCG supertags as structured sequences rather than atomic labels, we can build more robust and flexible parsers. While the generation process is computationally more intensive than simple classification, the gains in handling rare and unseen categories are substantial. Future work will focus on optimizing decoding speed and exploring more sophisticated attention mechanisms to further bridge the gap between efficiency and linguistic depth.

Summary updated

Abstract

Previous CCG supertaggers usually predict categories using multi-class classification. Despite their simplicity, internal structures of categories are usually ignored. The rich semantics inside these structures may help us to better handle relations among categories and bring more robustness into existing supertaggers. In this work, we propose to generate categories rather than classify them: each category is decomposed into a sequence of smaller atomic tags, and the tagger aims to generate the correct sequence. We show that with this finer view on categories, annotations of different categories could be shared and interactions with sentence contexts could be enhanced. The proposed category generator is able to achieve state-of-the-art tagging (95.5% accuracy) and parsing (89.8% labeled F1) performances on the standard CCGBank . Further-more, its performances on infrequent (even unseen) categories, out-of-domain texts and low resource language give promising results on introducing generation models to the general CCG analyses.
BibTeX
@inproceedings{liu2021generating,
  author = {Yufang Liu and Tao Ji and Yuanbin Wu and Man Lan},
  title = {{Generating CCG Categories}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=r2fl628CAGq},
}
Figure