The Shape of Words - topological structure in natural language data

Stephen FitzTAG-ML 2022 · OpenReview

Abstract

This paper presents a novel method, based on the ideas from algebraic topology, for the analysis of raw natural language text. The paper introduces the notion of a word manifold - a simplicial complex, whose topology encodes grammatical structure expressed by the corpus. Results of experiments with a variety of natural and synthetic languages are presented, showing that the homotopy type of the word manifold is influenced by linguistic structure. The analysis includes a new approach to the Voynich Manuscript - an unsolved puzzle in corpus linguistics. In contrast to existing topological data analysis approaches, we do not rely on the apparatus of persistent homology. Instead, we develop a method of generating topological structure directly from strings of words.

Introduction

In the field of Natural Language Processing (NLP), words are typically conceptualized as points within high-dimensional vector spaces. While metric-based embeddings like BERT or word2vec capture semantic relationships through distance, they often overlook the intrinsic "shape" or topological structure of language. This research proposes a shift toward algebraic topology, introducing the word manifold: a simplicial complex built directly from raw text sequences. By analyzing the topology of these structures, we can identify grammatical and syntactic constraints as fundamental topological features, such as "holes" or "cavities" in various dimensions.

Methodology

The construction of a word manifold avoids the need for neural encoders or predefined distance metrics, relying instead on the discrete sequence of tokens. The process follows three primary stages:

  1. Skeleta Construction: Words are treated as 0-dimensional vertices (0-simplices). If a sequence of words (an n-gram) appears within a specific window size $k$, it forms a higher-dimensional face (an n-simplex).
  2. Simplicial Homology: The method defines boundary matrices that describe how these n-dimensional shapes are glued together. This captures the connectivity of the linguistic structure.
  3. Betti Numbers: By computing the rank of homology groups, we extract Betti numbers ($\beta_n$). These represent the count of connected components ($\beta_0$), 1D loops ($\beta_1$), and 2D voids ($\beta_2$).
Standard n-simplices (\(\Delta^1, \Delta^2, \Delta^3\)) used to construct the word manifold.

Results and Linguistic Analysis

The study applied this topological framework to a diverse set of corpora, including modern news articles in six languages, the undeciphered Voynich Manuscript, and synthetic baselines.

Topological Signatures

Natural languages exhibit distinct topological signatures. For instance, French was found to have more 1D structure ($\beta_1$) than English, while English exhibited more complex 2D features ($\beta_2$). Interestingly, the number of connected components ($\beta_0$) in natural English hovers around eight, strikingly correlating with the eight classical parts of speech defined in antiquity. Japanese emerged as a high-dimensional outlier, showing significantly more structure in dimension 3 than other studied languages.

Visualizations of word manifolds showing clusters of grammatical categories such as nouns, verbs, and modifiers.

The Voynich Manuscript

The Voynich Manuscript proved to be a significant outlier in lower dimensions. However, its dimension 1 topology most closely aligns with Russian, providing a new mathematical data point for researchers investigating the manuscript's linguistic origins.

Syntax as Topological Glue

Ablation studies using synthetic "broken" corpora demonstrated that syntax acts as a "glue" that closes topological holes. Shuffling word order (Permutation) caused an unnatural explosion of 1D holes, while random sampling (Zipf or Uniform) destroyed nearly all topological features. This confirms that the word manifold captures meaningful linguistic constraints rather than random noise.

Conclusion

This work demonstrates that natural language possesses a rich topological structure that can be mapped directly from raw text. While the computational cost of calculating Betti numbers for large corpora is high, the word manifold offers a powerful new tool for comparative linguistics and the analysis of mysterious scripts. Future work will explore the mapping between these manifolds and neural embeddings to further bridge the gap between algebraic topology and modern NLP.

Summary updated

Abstract

This paper presents a novel method, based on the ideas from algebraic topology, for the analysis of raw natural language text. The paper introduces the notion of a word manifold - a simplicial complex, whose topology encodes grammatical structure expressed by the corpus. Results of experiments with a variety of natural and synthetic languages are presented, showing that the homotopy type of the word manifold is influenced by linguistic structure. The analysis includes a new approach to the Voynich Manuscript - an unsolved puzzle in corpus linguistics. In contrast to existing topological data analysis approaches, we do not rely on the apparatus of persistent homology. Instead, we develop a method of generating topological structure directly from strings of words.
BibTeX
@inproceedings{fitz2022shape,
  author = {Stephen Fitz},
  title = {{The Shape of Words - topological structure in natural language data}},
  booktitle = {TAG-ML},
  year = {2022},
  url = {https://openreview.net/forum?id=6iSZAUmP6o},
}
Figure