Towards Equipping Transformer with the Ability of Systematic Compositionality

Chen Huang, Peixin Qin, Wenqiang Lei, Jiancheng Lv

Abstract

Current Transformer models often treat word representations as monolithic vectors, struggling with "systematic compositionality"—the human ability to understand novel combinations of familiar semantic units. We present the Compositionality-Aware Transformer (CAT), which explicitly decomposes words into discrete semantic primitives and recomposes them. By grounding these primitives in linguistic sememes, CAT significantly improves performance on compositional and out-of-distribution tasks while maintaining general language understanding.

Introduction

While Large Language Models (LLMs) demonstrate remarkable capabilities, they frequently falter on tasks requiring a deep understanding of how individual word meanings (primitives) combine. For instance, a model might understand "safety" and "officer" in isolation but fail to recognize the semantic overlap between "safety officer" and "security guard," or the critical difference between "water body" and "body water." This limitation stems from a lack of systematic compositionality. Standard Transformers lack an explicit mechanism to break down words into fundamental building blocks and reassemble them contextually.

Methodology: The Compositionality-Aware Transformer (CAT)

The CAT architecture enhances the standard Transformer encoder with two specialized modules designed to foster discrete semantic reasoning.

The CAT architecture featuring Multi-Primitive Composition and Representation Enhancement modules.

Multi-Primitive Composition Module

This module transforms contextual word representations into a set of discrete "primitives" drawn from a trainable codebook. Unlike standard vector quantization, CAT employs a "soft" K-means approach with sparse attention to select a dynamic set of primitives. This allows the model to represent complex meanings through combinations of simpler, discrete units.

Representation Enhancement Module

To ensure that the model retains specific sentence context while benefiting from structured understanding, this module uses self-attention to fuse the original contextual representation ($h_{cont}$) with the new compositional representation ($h_{comp}$). The resulting mixed representation ($h_{mix}$) provides a more robust basis for downstream tasks.

Training Objectives

CAT is trained using two novel pre-training tasks:

  1. Guided Decomposition: Leverages OpenHowNet (a database of "sememes") to supervise the model, teaching it to map codebook primitives to real-world semantic units.
  2. Semantics Composition: A multi-objective loss function ensures compositional representations are consistent and sufficient for downstream tasks while filtering out irrelevant statistical noise.

Experimental Results

CAT was evaluated across a range of benchmarks focusing on compositionality, generalization, and general language understanding.

Compositional Superiority

As shown in the table below, CAT significantly outperforms baseline models like BERT and RoBERTa on compositionality-focused datasets. Notably, on the challenging BiRD-ABBA dataset—which requires distinguishing between phrases with identical words in different orders—CAT achieved a correlation score of 7.19, a substantial improvement over BERT's 1.64.

Performance comparison on adversarial (PAWS) and compositional correlation (BiRD) tasks.

Generalization and Robustness

CAT demonstrated superior out-of-distribution (OOD) generalization, with an average improvement of +1.85% over RoBERTa across various OOD datasets (e.g., IMDB, SST2, MNLI). Furthermore, the model showed enhanced adversarial robustness, outperforming BERT by 3.09% on SQuAD-adversarial. This suggests that decomposing words into core primitives helps the model ignore irrelevant "distractor" information.

General Language Understanding

Importantly, the introduction of compositionality-aware modules does not come at the cost of general performance. CAT maintains a comparable average score on the GLUE benchmark (76.16%) relative to BERT (76.07%), proving its versatility.

Conclusion

The Compositionality-Aware Transformer (CAT) addresses a fundamental weakness in current language models by introducing explicit semantic decomposition and recomposition. By forcing the model to operate in a discrete latent space grounded in linguistic primitives, CAT achieves superior performance on compositional tasks and better generalization to unseen contexts. Future work will explore scaling these techniques to larger generative models and optimizing codebook structures.

Summary updated

Abstract

One of the key factors in language productivity and human cognition is the ability of Systematic Compositionality, which refers to understanding composed, unseen examples of seen primitives. However, recent evidence reveals that the Transformers have difficulty in generalizing the composed context based on the seen primitives. To this end, we take the first step to propose a compositionality-aware Transformer called CAT and two novel pre-training tasks to facilitate the systematic compositionality. We tentatively provide a successful implementation of a multi-layer CAT on the basis of the especially popular BERT. The experimental results demonstrate that CAT outperforms baselines on compositionality-aware tasks with minimal impact on effectiveness on standardized language understanding tasks.
BibTeX
@inproceedings{huang2024towards,
  author = {Chen Huang and Peixin Qin and Wenqiang Lei and Jiancheng Lv},
  title = {{Towards Equipping Transformer with the Ability of Systematic Compositionality}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i16.29788},
}
Figure