Towards Equipping Transformer with the Ability of Systematic Compositionality

Chen Huang, Peixin Qin, Wenqiang Lei, Jiancheng Lv

Introduction

A hallmark of human intelligence is systematic compositionality: the capacity to understand and produce an infinite variety of novel expressions by combining a finite set of known components. For instance, a person who understands the concepts of "water" and "body" can immediately distinguish between a "water body" (a geographic feature) and "body water" (a physiological fluid).

Despite their remarkable success, modern Transformer-based models like BERT and even large-scale LLMs often struggle with this principle. They tend to treat phrases as holistic chunks rather than structured compositions, leading to failures when word order is permuted or when encountering unfamiliar combinations of familiar terms. This paper introduces the Compositionality-Aware Transformer (CAT), an architecture designed to explicitly model the compositional nature of language.

The CAT Framework

The CAT framework enhances the standard Transformer encoder by integrating two specialized modules that force the model to learn semantic building blocks.

Figure 1: The CAT architecture featuring Multi-Primitive Composition and Representation Enhancement.
  1. Multi-Primitive Composition Module: This module decomposes a word's contextual representation ($h_{cont}$) into multiple discrete "primitives" ($e_i$) drawn from a trainable codebook. These primitives are conceptually similar to "sememes"—the smallest units of semantic meaning. By recomposing these primitives into a compositional representation ($h_{comp}$), the model is forced to learn the underlying logic of concept combination rather than just memorizing patterns.
  2. Representation Enhancement Module: To maintain the model's general linguistic capabilities, a self-attention mechanism fuses the original contextual representation with the new compositional representation, resulting in a final mixed representation ($h_{mix}$).

The model is trained using two novel pre-training tasks: Guided Decomposition, which uses the OpenHowNet knowledge base to supervise sememe mapping, and Semantics Composition, which employs loss functions to ensure the composed representations remain semantically consistent with the input.

Experimental Evaluation

The authors evaluated CAT against standard baselines (BERT, RoBERTa, DistilBERT) across a variety of tasks emphasizing structural understanding and general performance.

Compositional and Adversarial Robustness

The core of the evaluation focused on the model's ability to handle word swaps and relatedness.

  • PAWS: A dataset for paraphrase identification with high word overlap.
  • BiRD & BiRD-ABBA: The authors introduced BiRD-ABBA, a subset of the Bigram Relatedness Dataset where phrase pairs consist of the exact same words in different orders (e.g., "Water body" vs. "Body water").
Table 1: Performance on compositional (BiRD) and adversarial (PAWS) benchmarks.

As shown in Table 1, the compositional variant of CAT ($CAT_{comp}$) significantly outperforms BERT on the challenging BiRD-ABBA task, achieving a correlation score of 7.19 compared to BERT's 1.64. On the PAWS adversarial dataset, CAT achieved 90.33% accuracy, demonstrating superior robustness to word-order permutations.

General Performance and Robustness

Critically, CAT does not sacrifice general language understanding. On the GLUE benchmark, CAT maintained an average score of 76.16, comparable to BERT's 76.07. Furthermore, CAT showed improved robustness in out-of-distribution (OOD) scenarios, such as IMDB and Amazon review sentiment analysis, and outperformed BERT by 3.09% on adversarial reading comprehension (SQuAD-adversarial).

Discussion and Conclusion

The results suggest that forcing models to use a discrete codebook of primitives acts as a powerful regularizer, preventing the "memorization" of phrase patterns. Interestingly, the authors found that even ChatGPT struggled with the BiRD-ABBA task (scoring 23.28), suggesting that scale alone does not solve the problem of systematic compositionality.

While CAT shows great promise, future work is needed to scale this architecture to billion-parameter decoder-only models and to reduce the dependency on human-curated knowledge bases like OpenHowNet. Nonetheless, CAT represents a significant step toward equipping AI with the structural reasoning capabilities inherent in human language.

Summary updated

BibTeX
@inproceedings{huang2024towards,
  author = {Chen Huang and Peixin Qin and Wenqiang Lei and Jiancheng Lv},
  title = {{Towards Equipping Transformer with the Ability of Systematic Compositionality}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i16.29788},
}
Figure