Abstract
This paper proposes a mathematical framework for "universal semantics" by modeling text as a Markov process and analyzing the statistical timing of recurring word patterns. By extracting language-independent numerical fingerprints (eigenvalues) from these patterns, we demonstrate that machines can perform complex tasks like word translation and question-answering using only a single document, without needing external dictionaries or massive pre-training.
Introduction
A fundamental challenge in Natural Language Processing (NLP) is defining the "meaning" of a word in a way that a computer can understand regardless of the language. Traditional methods often rely on data-hungry models or hand-curated knowledge bases. Furthermore, meaning is often entangled with language-specific grammar and syntax. This research seeks a universal mechanism of semantics based on numerical characteristics that transcend specific written or spoken forms, extracting structure directly from the statistical patterns within a text.
Methodology: The Time Structure of Meaning
The core hypothesis is that the temporal distribution of recurring language patterns—how often and in what contexts words reappear—serves as a proxy for semantic meaning.
Word Patterns and Recurrence
Rather than individual words, the model utilizes "word patterns" ($W_i$), which group morphologically related content words (e.g., {happy, happily, happiness}). The model measures the distance between occurrences of the same pattern (recurrence time, $L_{ii}$) and the distance to reach a new pattern (hitting time, $L_{ij}$).
Topicality vs. "Poissonian Banality"
A key insight is that non-topical words (like "the" or "of") recur randomly, following a Poisson process. In contrast, topical words (like "Jane" or "pride") show significant deviations from this randomness, clustering in specific parts of the text to encode semantic connectivity.
Language-Independent Semantic Fingerprints (LISF)
By approximating text generation as a stochastic process on a web of word patterns, the authors discovered that the spectrum (eigenvalues) of the resulting Markov matrices is nearly invariant across different languages. These eigenvalues form the LISF of a word, derived from its "semantic clique"—a localized neighborhood of closely related concepts.
Key Results
The researchers evaluated their model using parallel corpora (e.g., Pride and Prejudice in 14 languages) and the WikiQA benchmark.
Automated Word Translation
By matching LISF vectors across languages, the model achieved high precision in translating topical words between English and 13 other languages, even across different language families (e.g., English to Finnish or Russian), without any prior cross-lingual data.
Question Answering
On the WikiQA dataset, the model's associative reasoning approach—modeled as navigation through semantic cliques—performed on par with established benchmarks like CNN-based models, despite having no pre-trained word embeddings or external knowledge bases.
Discussion and Limitations
While the model can extract meaningful representations from a single document, it requires texts of moderate length to gather sufficient statistical samples. Currently, the model focuses on long-range semantic features and ignores short-range syntactic structures. Future work aims to combine these scales and explore Markov Random Fields to better capture multi-topic interactions.
Summary updated