What company do words keep? Revisiting the distributional semantics of J.R. Firth & Zellig Harris

Mikael Brunila, Jack LaVioletteNAACL 2022 · DOI

Abstract

Modern Natural Language Processing (NLP) is built upon the foundational mantra: "You shall know a word by the company it keeps." While this principle is frequently attributed to J.R. Firth, contemporary models often implement a version of context more aligned with the mathematical structuralism of Zellig Harris. This paper, based on the analysis by Mikael Brunila and Jack LaViolette (NAACL 2022), explores the historical and theoretical divergence between Firthian and Harrisian semantics. It argues that modern NLP has suffered from "theoretical amnesia," conflating two distinct ideologies and ultimately hitting a ceiling by treating language as a closed, decontextualized system. By revisiting Firth’s "context of situation," the authors propose a roadmap for moving beyond "stochastic parrots" toward socially and culturally grounded models.

Introduction

In the current era of Large Language Models (LLMs), the Distributional Hypothesis—the idea that words occurring in similar contexts share similar meanings—is treated as an empirical law. From the early days of Word2Vec to the sophisticated architectures of GPT-4, the field has relied on the "company words keep" to derive semantic representations. However, as Brunila and LaViolette argue, the definition of "company" has become increasingly narrow.

In their theoretical review, the authors suggest that the NLP community has largely ignored the social and anthropological roots of distributionalism. By performing a close reading of the foundational texts of British linguist J.R. Firth and American linguist Zellig Harris, they reveal a fundamental tension between language as a mathematical form and language as a social practice.

The Divergent Roots: Harris vs. Firth

While textbooks often cite Firth and Harris as a unified front, their philosophies were radically different.

Harris’s Distributional Structuralism

Zellig Harris, a mathematical formalist and mentor to Noam Chomsky, viewed language as a "detached pattern." For Harris, meaning was a byproduct of the mathematical constraints of how linguistic forms (phonemes, morphemes, and words) co-occur. In this view, language is a self-contained system. If you have enough data and a powerful enough model to map the internal distributions, you can theoretically capture the essence of the language. This "Harrisian" perspective is the direct ancestor of modern NLP, where context is defined as a sliding window of neighboring tokens in a massive, decontextualized corpus.

Firth’s Contextual Semantics

In contrast, J.R. Firth, influenced by the anthropologist Bronisław Malinowski, argued that meaning is "polysystemic" and inseparable from the "context of situation." For Firth, a word does not just keep company with other words; it keeps company with the world. A "context of situation" includes the participants, their social roles, their non-verbal actions, and the physical objects present during an utterance. Firth rejected the idea of a single, universal language model, instead advocating for the study of "restricted languages"—specialized registers like the language of a specific marketplace or a scientific discipline—where words take on distinct distributional patterns based on social utility.

The Problem of "Theoretical Amnesia"

The authors argue that by following the Harrisian path, NLP has reached a point of diminishing returns regarding true "understanding." Modern models are essentially "stochastic parrots" because they lack access to the non-linguistic variables that Firth deemed essential. When we train a model on a massive, undifferentiated crawl of the internet, we lose the social, historical, and situational nuances that define human communication. We treat meaning as a monolith rather than a dynamic negotiation between social actors.

A Roadmap for Firthian NLP

To bridge the gap between formalist models and social reality, the paper proposes two conceptual strategies for the future of the field:

  1. Comparative Stratification: Rather than training monolithic models on all available text, researchers should "stratify" data by time, space, and social community. By comparing how the "company words keep" shifts across different strata (e.g., how the word "cell" functions in a biological text versus a legal document), models can begin to capture the situational nature of meaning.
  2. Syntagmatic Extension: This involves explicitly modeling non-linguistic variables as part of the vector space. By treating speaker demographics, geographic location, and even accompanying sensory data (like images or timestamps) as equal participants in the distributional calculation, we can move closer to Firth’s vision of a socially grounded semantics.

Conclusion

The transition from the "massive corpus" era to a "culturally grounded" era requires a return to linguistic roots. While Harris provided the mathematical framework that allowed NLP to scale, Firth provides the theoretical depth necessary for models to navigate the complexities of human society. By recognizing that meaning is not just a pattern of tokens but a reflection of the human world, the field can move toward a more robust and nuanced understanding of language.

Summary updated

BibTeX
@inproceedings{brunila2022what,
  author = {Mikael Brunila and Jack LaViolette},
  title = {{What company do words keep? Revisiting the distributional semantics of J.R. Firth \& Zellig Harris}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.327},
}
Figure