A case for deep learning in semantics

Christopher PottsarXiv 2018 · arXiv

Abstract

Traditional formal semantics, rooted in intensional logic and mathematical functions, has long maintained a divide between theoretical linguistics and the empirical successes of Natural Language Processing (NLP). This paper argues for a paradigm shift: the adoption of deep learning (DL) frameworks within semantic theory. By replacing logical functions with $n$-dimensional arrays (tensors) and learned composition rules, semanticists can better capture the usage-based nuances of natural language. While DL models currently struggle with certain logical properties, the integration of linguistic theory into neural architectures offers a path toward a more robust, data-driven science of meaning.

Introduction

For decades, the foundation of formal semantics has been the idea that the meaning of a word or sentence is a mathematical function. While this approach is elegant and precise, it has created a significant rift between theoretical research and the rapidly advancing field of NLP. As deep learning researchers make strides in "compositional semantics"—the study of how word meanings combine—they often do so without the insights of linguistic theory. Conversely, many semanticists dismiss DL as uninterpretable or disconnected from logic.

Christopher Potts argues that this isolation is a risk to the field. If semanticists do not engage with DL, theoretical semantics may become a branch of mathematics disconnected from how language is actually acquired and used. This paper explores the case for "DL Semantics," a synthesis that uses the power of neural networks to handle the messy reality of language while using formal theory to guide logical architecture.

From Functions to Tensors

The proposed shift involves a fundamental change in the "building blocks" of meaning. In traditional semantics, meanings are functions; in DL semantics, meanings are $n$-dimensional arrays, such as vectors, matrices, or tensors.

Learned Composition

Instead of simple function application (e.g., $f(x)$), meanings in a DL framework are combined using learned weight matrices ($W$) and non-linear activation functions (such as $tanh$). This allows the model to learn the specific rules of combination from data rather than having them hard-coded by a linguist.

Recursive Structure

Recursive Neural Networks (TreeRNNs) serve as a bridge between these two worlds. These models adhere to the "Principle of Compositionality" by building the meaning of a phrase from its constituent parts, yet they remain flexible enough to capture nuances that strict logical models often miss.

Key Insights and Theoretical Implications

The Functionalist View of Meaning

Invoking philosopher David Lewis’s advice—"In order to say what a meaning is, we may first ask what a meaning does"—Potts suggests that if vectors can successfully predict usage, sentiment, and entailment, then vectors are valid representations of meaning. This functionalist approach allows semantics to move beyond idealized models.

Blurring the Semantics-Pragmatics Divide

Traditional theory attempts to separate literal meaning (semantics) from contextual use (pragmatics). However, because DL models learn from actual usage data, they naturally blur this line. They capture frequency biases, social refinements, and argumentative structures that logic-based models typically ignore.

Generalization vs. Compositionality

In machine learning, the primary goal is generalization to unseen data rather than adherence to strict compositional rules. Potts suggests that strict compositionality might actually be a "restrictive strategy" that hinders a model's ability to handle the complexities of natural language.

Evidence and Challenges

Qualitative Proof of Concept

Research into sentiment composition (e.g., the Stanford Sentiment Treebank) demonstrates that learned weight matrices can correctly model complex interactions. For instance, a DL model can learn that "not terrible" is less negative and less intense than "terrible" alone. Similarly, models can capture the argumentative weight of coordinators like "but," where the second clause typically carries more importance—a feat difficult to achieve with pure logic.

The "Monotyped" and Logical Challenge

A significant hurdle for current DL models is that they are often "monotyped," treating every word (from "zebra" to "every") as a vector of the same shape. This is insufficient for representing complex logic like quantification. Furthermore, even effective DL models often fail to capture specific logical properties like monotonicity, highlighting the need for linguistic guidance in model design.

Conclusion

The future of semantics lies in a synthesis of formal theory and deep learning. While DL provides the tools to handle the lexicon and usage-based nuances, formal semantics provides the necessary framework for human-level reasoning and logical structure. Semanticists are encouraged to insert themselves into the DL discourse to ensure the field remains relevant to the study of language as it is truly used.

Summary updated

Abstract

Pater's target article builds a persuasive case for establishing stronger ties between theoretical linguistics and connectionism (deep learning). This commentary extends his arguments to semantics, focusing in particular on issues of learning, compositionality, and lexical meaning.
BibTeX
@misc{potts2018case,
  author = {Christopher Potts},
  title = {{A case for deep learning in semantics}},
  year = {2018},
  eprint = {1809.03068},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1809.03068},
}
Figure