Abstract
Current Transformer models often treat word representations as monolithic vectors, struggling with "systematic compositionality"—the human ability to understand novel combinations of familiar semantic units. We present the Compositionality-Aware Transformer (CAT), which explicitly decomposes words into discrete semantic primitives and recomposes them. By grounding these primitives in linguistic sememes, CAT significantly improves performance on compositional and out-of-distribution tasks while maintaining general language understanding.
Introduction
While Large Language Models (LLMs) demonstrate remarkable capabilities, they frequently falter on tasks requiring a deep understanding of how individual word meanings (primitives) combine. For instance, a model might understand "safety" and "officer" in isolation but fail to recognize the semantic overlap between "safety officer" and "security guard," or the critical difference between "water body" and "body water." This limitation stems from a lack of systematic compositionality. Standard Transformers lack an explicit mechanism to break down words into fundamental building blocks and reassemble them contextually.
Methodology: The Compositionality-Aware Transformer (CAT)
The CAT architecture enhances the standard Transformer encoder with two specialized modules designed to foster discrete semantic reasoning.
Multi-Primitive Composition Module
This module transforms contextual word representations into a set of discrete "primitives" drawn from a trainable codebook. Unlike standard vector quantization, CAT employs a "soft" K-means approach with sparse attention to select a dynamic set of primitives. This allows the model to represent complex meanings through combinations of simpler, discrete units.
Representation Enhancement Module
To ensure that the model retains specific sentence context while benefiting from structured understanding, this module uses self-attention to fuse the original contextual representation ($h_{cont}$) with the new compositional representation ($h_{comp}$). The resulting mixed representation ($h_{mix}$) provides a more robust basis for downstream tasks.
Training Objectives
CAT is trained using two novel pre-training tasks:
- Guided Decomposition: Leverages OpenHowNet (a database of "sememes") to supervise the model, teaching it to map codebook primitives to real-world semantic units.
- Semantics Composition: A multi-objective loss function ensures compositional representations are consistent and sufficient for downstream tasks while filtering out irrelevant statistical noise.
Experimental Results
CAT was evaluated across a range of benchmarks focusing on compositionality, generalization, and general language understanding.
Compositional Superiority
As shown in the table below, CAT significantly outperforms baseline models like BERT and RoBERTa on compositionality-focused datasets. Notably, on the challenging BiRD-ABBA dataset—which requires distinguishing between phrases with identical words in different orders—CAT achieved a correlation score of 7.19, a substantial improvement over BERT's 1.64.
Generalization and Robustness
CAT demonstrated superior out-of-distribution (OOD) generalization, with an average improvement of +1.85% over RoBERTa across various OOD datasets (e.g., IMDB, SST2, MNLI). Furthermore, the model showed enhanced adversarial robustness, outperforming BERT by 3.09% on SQuAD-adversarial. This suggests that decomposing words into core primitives helps the model ignore irrelevant "distractor" information.
General Language Understanding
Importantly, the introduction of compositionality-aware modules does not come at the cost of general performance. CAT maintains a comparable average score on the GLUE benchmark (76.16%) relative to BERT (76.07%), proving its versatility.
Conclusion
The Compositionality-Aware Transformer (CAT) addresses a fundamental weakness in current language models by introducing explicit semantic decomposition and recomposition. By forcing the model to operate in a discrete latent space grounded in linguistic primitives, CAT achieves superior performance on compositional tasks and better generalization to unseen contexts. Future work will explore scaling these techniques to larger generative models and optimizing codebook structures.
Summary updated