Recursive Neural Networks with Bottlenecks Diagnose (Non-)Compositionality

Verna Dankers, Ivan Titov

Abstract

A recent line of work in NLP focuses on the (dis)ability of models to generalize compositionally for artificial languages. However, natural language data is not strictly compositional. Quantifying the compositionality of data is challenging, especially for longer utterances. We use recursive neural models (Tree-LSTMs) with bottlenecks that limit information transfer between nodes. We demonstrate that comparing data representations in models with and without bottlenecks produces a compositionality metric. Applied to arithmetic expressions and sentiment classification, we show that compression impacts non-compositional examples disproportionately. The resulting Bottleneck Compositionality Metric (BCM) distinguishes compositional from non-compositional samples, yielding a dataset-wide compositionality ranking.

Introduction

In Natural Language Processing (NLP), "compositional generalization" is the gold standard: the ability to understand complex expressions by combining the meanings of their parts. However, natural language is rarely perfectly compositional. Idioms, sarcasm, and context-dependent meanings (like "kick the bucket" or "the ruler") mean that the whole is often more than the sum of its parts.

Ambiguous examples in natural language demonstrate how interpretation can change based on context, a hallmark of non-compositionality.

A new study by Dankers and Titov (EMNLP 2022) introduces the Bottleneck Compositionality Metric (BCM). This metric diagnoses how compositional a piece of data is by intentionally "squeezing" the information flow within a neural network.

The Challenge of Non-Compositionality

Most models assume local compositionality, where a phrase's meaning is determined by its immediate constituents. But non-compositional language—like idioms or sarcasm—requires the model to "memorize" specific patterns or carry global context through the tree. Quantifying this has been difficult. Previous methods like Tree Reconstruction Error (TRE) are theoretically sound but hard to scale. BCM offers a task-agnostic, practical alternative.

Methodology: The Bottleneck Approach

The researchers utilized Recursive Neural Networks (Tree-LSTMs), which process text according to hierarchical syntactic structures. They compared a standard "base" Tree-LSTM to a "bottlenecked" version. Three types of bottlenecks were tested:

  1. Deep Variational Information Bottleneck (DVIB): Penalizes the information (in bits) passed through the network.
  2. Dropout: Forces robust, distributed representations by randomly zeroing neurons.
  3. Hidden Dimensionality: Reduces layer size to force compression.

The BCM is calculated by comparing the representations of the base and bottlenecked models. If a bottleneck significantly alters an example's representation, it suggests the example relied on non-local, non-compositional information that could not pass through the restricted "squeeze."

Experimental Results

The metric was evaluated on two tasks: synthetic arithmetic (with manually injected "exceptions") and the Stanford Sentiment Treebank (SST-5).

Non-Compositionality is "Expensive"

The experiments revealed that non-compositional examples are the first to fail under pressure. In the arithmetic task, "exceptions" (where a number's meaning changed based on its tree position) showed a massive increase in Mean Squared Error (MSE) as the bottleneck strength (beta) increased.

In the arithmetic task, non-compositional "exceptions" show a dramatic increase in error (MSE) as the bottleneck strength increases, compared to regular compositional examples.

In natural language (SST-5), the BCM successfully ranked "positive" and "negative" patterns as highly compositional, while identifying sarcasm, irony, and mixed sentiment as highly non-compositional.

Key Insights

  • Bottlenecks Enforce Locality: Restricting capacity forces a model to process meaning locally at each node, as it can no longer afford to carry the identity of distant leaf nodes up the tree.
  • Data Efficiency: Training on highly compositional examples (as ranked by BCM) leads to better performance in data-scarce scenarios than training on random or non-compositional samples.
  • Performance Gap: Standard LSTMs saw an 11-percentage-point drop in accuracy when tested on a non-compositional split of sentiment data, highlighting the objective difficulty of these samples.

Conclusion

While currently tied to Tree-LSTMs and requiring syntactic trees, BCM provides a powerful diagnostic tool. Future work could integrate these bottlenecks into Transformer architectures or use BCM to develop "compositionality-based curricula," training models on simpler, compositional language before introducing the expensive nuances of non-compositional speech.

Summary updated

Abstract

A recent line of work in NLP focuses on the (dis)ability of models to generalise compositionally for artificial languages.However, when considering natural language tasks, the data involved is not strictly, or locally, compositional.Quantifying the compositionality of data is a challenging task, which has been investigated primarily for short utterances.We use recursive neural models (Tree-LSTMs) with bottlenecks that limit the transfer of information between nodes.We illustrate that comparing data’s representations in models with and without the bottleneck can be used to produce a compositionality metric.The procedure is applied to the evaluation of arithmetic expressions using synthetic data, and sentiment classification using natural language data.We demonstrate that compression through a bottleneck impacts non-compositional examples disproportionatelyand then use the bottleneck compositionality metric (BCM) to distinguish compositional from non-compositional samples, yielding a compositionality ranking over a dataset.
BibTeX
@inproceedings{dankers2022recursive,
  author = {Verna Dankers and Ivan Titov},
  title = {{Recursive Neural Networks with Bottlenecks Diagnose (Non-)Compositionality}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=iBBHZy33l1},
}
Figure