AND does not mean OR: Using Formal Languages to Study Language Models' Representations

Aaron Traylor, Roman Feiman, Ellie PavlickACL 2021 · DOI

TL;DR

This paper investigates whether language models (LMs) can learn the meaning of logical operators solely from their form, using a synthetic propositional logic language. The authors found that even with carefully constructed training data, LMs struggle to differentiate between semantically similar but syntactically distinct logical operators, suggesting limitations in how LMs exploit distributional signals for meaning.

What Problem Does This Paper Address?

A central question in natural language processing (NLP) is whether neural language models, trained exclusively on the form of language (text), can truly understand its meaning. While some argue that LMs are fundamentally limited because they lack direct access to real-world grounding, others suggest that strong correlations between linguistic form and meaning in natural language might allow LMs to infer meaning indirectly. This paper tackles a specific aspect of this debate: can LMs learn to differentiate meanings of logical operators (like AND, OR, NOT) when these operators have distinct truth conditions but can be syntactically similar? The authors aim to understand what kinds of "semantic transparency" in training data are necessary for LMs to achieve this differentiation.

Methods

The researchers created a controlled environment using propositional logic, where both the "form" (strings of symbols) and "meaning" (truth conditions) are precisely defined. They generated four types of synthetic training corpora, each with increasing levels of "semantic transparency":

  1. Syntactic Constraint: Sentences are simply well-formed according to a grammar.
  2. Truthfulness Constraint: Sentences must be true in at least one possible world.
  3. Informativity Constraint: Sentences must convey information by being true in a set of "target worlds" and false in a set of "alternative worlds."
  4. Explicit Grounding: Sentences are paired with explicit truth assignments for all variables (this is not a realistic corpus generation method but serves as an upper bound).

They then trained standard LMs (LSTMs and Transformers) on these corpora and evaluated their ability to differentiate the representations of logical operators (specifically AND, OR, and NOT) using a nearest-neighbors probing classifier. This classifier checks if the learned embeddings for operators with different meanings cluster separately.

Datasets

The paper introduces four novel synthetic datasets based on propositional logic, each designed to vary in its "semantic transparency." These datasets are generated using a probabilistic context-free grammar. The datasets are:

  • Syntactic Constraint Dataset
  • Truthfulness Constraint Dataset
  • Informativity Constraint Dataset
  • Explicit Grounding Dataset

Each dataset consists of 100,000 training sentences and 1,000 validation sentences. The number of non-reserved symbols was set to 5,000, and each logical operator had 5 "synonyms."

Key Insights and Ideas

  • Form vs. Meaning is Tricky: Even in a highly controlled, synthetic environment with precise definitions of form and meaning, LMs struggle to reliably distinguish between semantically distinct logical operators that are syntactically similar.
  • Syntactic Clues Help, But Aren't Enough: While LMs can easily differentiate unary operators (like NOT) from binary operators (like AND/OR) due to syntactic differences, they fail to consistently distinguish between binary operators like AND and OR.
  • Transformer Architecture Shows Promise: Transformer models, compared to LSTMs, appear to be slightly better at picking up on subtle semantic signals, even from less transparent datasets like the "Truthfulness" constraint.
  • "Grounding" Data is Key (But Unrealistic): The "Explicit Grounding" dataset, which provides direct truth assignments, yields the most distinguishable operator representations. However, this method is not representative of how natural language corpora are formed.
  • Limitations of Lexical Representations: The study focuses on differentiating operators at the lexical level (i.e., in word embeddings). The authors acknowledge that semantic distinctions might be encoded elsewhere in the model, beyond just embeddings, which their current analysis doesn't fully capture.

Key Results

  • Operator Differentiation: In nearly all experimental settings, models could differentiate the unary operator (¬) from the binary operators (∧, ∨). However, differentiating between ∧ and ∨ was significantly harder.
  • Best Performance: The "Explicit Grounding" dataset consistently yielded the highest accuracy for operator differentiation across models.
  • Worst Performance: The "Syntactic Constraint" dataset generally resulted in the least distinguishable operator representations.
  • Model Differences: Transformer models generally outperformed LSTMs, especially on the "Truthfulness" dataset, suggesting a greater capacity for learning from weaker semantic signals.
  • Accuracy: For the challenging task of differentiating ∧ and ∨, even the best-performing models (Small Transformer on Explicit Grounding) achieved accuracies only slightly above random chance (33%), indicating a failure to fully differentiate meaning.

Limitations and Future Work

  • Focus on Lexical Representations: The study primarily analyzes operator differentiation at the embedding layer. The authors suggest that future work should explore whether semantic distinctions are encoded in compositional semantics or other parts of the model.
  • Synthetic Data vs. Natural Language: The synthetic nature of the data, while controlled, might not fully capture the complexities, noise, and ambiguity present in real-world language corpora. Further research is needed to understand how these findings translate to natural language settings.
  • Informativity Dataset Comparability: The authors note that the experimental setup for the "Informativity" dataset (shorter sentences) makes direct comparison to other datasets challenging, requiring further study.
  • Model Scale: The study used relatively small models. Exploring larger models could reveal different capabilities for learning semantic distinctions.

Summary updated

BibTeX
@inproceedings{traylor2021does,
  author = {Aaron Traylor and Roman Feiman and Ellie Pavlick},
  title = {{AND does not mean OR: Using Formal Languages to Study Language Models' Representations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-short.21},
}
Figure