Abstract: A recent line of work in NLP focuses on the (dis)ability of models to generalise compositionally for artificial languages. However, when considering natural language tasks, the data involved is not strictly, or locally, compositional. Quantifying the compositionality of data is a challenging task, which has been investigated primarily for short utterances. We use recursive neural models (Tree-LSTMs) with bottlenecks that limit the transfer of information between nodes. We illustrate that comparing data's representations in models with and without the bottleneck can be used to produce a compositionality metric. The procedure is applied to the evaluation of arithmetic expressions using synthetic data, and sentiment classification using natural language data. We demonstrate that compression through a bottleneck impacts non-compositional examples disproportionately and then use the bottleneck compositionality metric (BCM) to distinguish compositional from non-compositional samples, yielding a compositionality ranking over a dataset.
Introduction
A fundamental question in linguistics and AI is how much of a sentence's meaning is derived from its individual parts versus its broader context. While the principle of compositionality—that the meaning of a complex expression is determined by the meanings of its constituent expressions and the rules used to combine them—is a cornerstone of language processing, natural language is often "quasi-compositional." It contains idioms, sarcasm, and context-dependent nuances that defy simple algebraic combination. This paper introduces a method to quantify these deviations using information bottlenecks in recursive neural networks.
The Bottleneck Compositionality Metric (BCM)
The authors propose the Bottleneck Compositionality Metric (BCM) to distinguish between local processing (building meaning from parts) and global memorization (relying on specific context). The core idea is that compositional meaning is information-efficient. For example, to compute the result of an addition, a model only needs to pass the sum up the tree. However, to handle a non-compositional exception, a model might need to pass the raw identity of specific tokens to higher levels, requiring higher informational capacity.
By introducing bottlenecks—such as the Deep Variational Information Bottleneck (DVIB), Dropout, or reduced hidden dimensionality—into Tree-LSTMs, the researchers can "starve" the model of the capacity needed for expensive non-compositional strategies. The BCM is calculated by comparing the internal representations of a standard "base" model with those of a "bottlenecked" model. A large difference indicates that the example relies on non-compositional information that the bottleneck successfully filtered out.
Experimental Setup and Datasets
The method was tested on two distinct datasets to evaluate its robustness across synthetic and natural language:
- Synthetic Arithmetic: A dataset of mathematical expressions where "exceptions" were manually introduced. For instance, a rule might dictate that the number '0' takes a different value if it appears in a specific subtree, forcing the model to track its position globally rather than just its value locally.
- Stanford Sentiment Treebank (SST-5): A natural language dataset of movie reviews. This allowed the researchers to see if BCM could identify linguistic phenomena like sarcasm or irony.
Key Results
The BCM successfully identified non-compositional elements in both settings. In the arithmetic task, the metric accurately ranked the "exception" cases as the least compositional. In the natural language task, phrases involving sarcasm or world knowledge (e.g., "a thriller without thrills") were ranked as least compositional, while simple, direct sentiment phrases were ranked as most compositional.
Furthermore, the study demonstrated the practical utility of this ranking. When training on limited data (10% of the total), using the most compositional examples as identified by BCM led to an 11% accuracy improvement over using the least compositional ones. This suggests that compositional data is more "efficient" for learning generalizable rules.
Conclusion and Future Work
While the current implementation relies on the availability of high-quality parse trees and a strict definition of local compositionality, it provides a powerful, task-agnostic tool for diagnosing how models process language. Future work may extend this bottleneck approach to non-recursive architectures like Transformers, potentially offering deeper insights into the "black box" of modern NLP models.
Summary updated