Holographic CCG Parsing

Ryosuke Yamaki, Tadahiro Taniguchi, Daichi MochihashiACL 2023 · DOI

Introduction

Combinatory Categorial Grammar (CCG) is a linguistically motivated framework that provides a transparent interface between syntax and semantics. While modern neural CCG parsers achieve high accuracy using architectures like Transformers or LSTMs, they often treat phrase composition as an implicit "black-box" process. This lack of explicit structure makes it difficult to interpret how individual word representations combine into higher-level phrasal meanings.

Previous attempts to explicitly model these dependencies, such as Compositional Vector Grammars (CVG), suffered from parameter explosion due to the need for unique matrices for every grammar rule. This paper introduces a middle ground: a model that leverages the expressive power of high-dimensional vectors while strictly following recursive CCG rules using Holographic Embeddings (HolE).

Methodology: Holographic Composition

The core innovation lies in using circular correlation ($\star$) as a fixed compositional operator. Unlike standard neural layers, circular correlation is non-commutative ($a \star b \neq b \star a$) and non-associative, properties that are essential for capturing the directed and hierarchical nature of language.

The model architecture utilizes a RoBERTa encoder followed by recursive holographic composition to build span representations.

As shown in the architecture above, the model first generates word-level vectors using a Deep Contextualized Encoder (RoBERTa). These vectors are then recursively combined following a CCG derivation tree. To manage vector magnitudes during deep recursion, the authors implemented norm constraints in both real and complex vector spaces. The final span representations are fed into three specialized modules: a Span Existence Discriminator, a Phrase Category Classifier, and a Word Category Classifier (Supertagger).

Experimental Setup

The model was evaluated on the standard CCGbank dataset, derived from the Wall Street Journal corpus. The researchers employed a novel CKY-style span-based parsing algorithm that calculates scores for every possible word sequence to find the most likely grammatical structure.

Key Results

The holographic approach demonstrated that explicit mathematical composition does not come at the cost of performance. In fact, the model achieved a new state-of-the-art in supertagging accuracy.

Comparison of the holographic model against previous state-of-the-art CCG parsers and supertaggers.

As detailed in the table above, the model reached a supertagging accuracy of 96.60%, outperforming complex "black-box" architectures. The span-based parser achieved a labeled F-score (LF) of 92.67%, making it highly competitive with current Transformer-based systems.

Beyond raw metrics, the model's "decomposability" is a significant advantage. Because circular correlation is mathematically reversible, the model can perform syntactic infilling. In head-to-head tests against RoBERTa, the holographic model was significantly better at maintaining syntactic consistency, achieving a 96.31% match rate for the grammatical category of filled phrases, compared to 77.95% for RoBERTa.

Discussion and Future Work

While the results are promising, the model currently requires supervised data from CCGbank, limiting its applicability to languages with similar resources. Additionally, the current Python implementation of the span-based parser can be computationally intensive for very long sentences. Future work will focus on optimizing the parsing algorithm and extending the model to handle unary rules more elegantly. By bridging the gap between symbolic grammar and vector-based representations, holographic parsing opens new avenues for interpretable and decomposable natural language processing.

Summary updated

BibTeX
@inproceedings{yamaki2023holographic,
  author = {Ryosuke Yamaki and Tadahiro Taniguchi and Daichi Mochihashi},
  title = {{Holographic CCG Parsing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.15},
}
Figure