Abstract
We propose a theoretical framework within which information on the vocabulary of a given corpus can be inferred on the basis of statistical information gathered on that corpus. Inferences can be made on the categories of the words in the vocabulary, and on their syntactical properties within particular languages. Based on the same statistical data, it is possible to build matrices of syntagmatic similarity (bigram transition matrices) or paradigmatic similarity (probability for any pair of words to share common contexts). When clustered with respect to their syntagmatic similarity, words tend to group into sublanguage vocabularies, and when clustered with respect to their paradigmatic similarity, into syntactic or semantic classes. This method yields a soft clustering of the vocabulary into sublanguages which contribute to the generation of heterogeneous corpora. As an application, we show how multilingual texts can be visually segmented into linguistically homogeneous segments.
Introduction
Traditional language identification is often treated as a solved problem, provided large training sets are available for well-documented languages. However, real-world data is frequently "composite," featuring code-switching or shifts between formal prose and specialized jargon. Existing tools struggle when languages are mixed within the same document, when no prior models exist (unsupervised settings), or when languages are closely related and share significant vocabulary.
This paper introduces an unsupervised method to detect language and discourse boundaries within a single text by performing spectral analysis on word transition matrices. By treating a corpus as a Markov chain, we identify distinct "generative systems" and visualize where they shift, even in the absence of pre-existing dictionaries.
Methodology
The proposed framework utilizes spectral analysis of the text's internal structure through a mathematical pipeline:
- Matrix Construction: A context matrix ($C_k$) is built to count word adjacencies, focusing on syntagmatic similarity.
- Symmetrization: A symmetric neighborhood matrix ($W$) is defined to avoid reading direction bias.
- Markov Chain Modeling: The matrix is normalized to create a transition matrix ($P = D^{-1}W$). The text is viewed as a random walk through a graph where words are nodes.
- Eigenvector Decomposition: Calculating the eigenvectors of the transition matrix identifies the principal axes of word distribution.
- Soft Clustering: This provides a probability distribution rather than a binary classification. A word's position on an eigenvector indicates its likelihood of belonging to a specific "generative system."
As illustrated above, individual languages or discourses can be modeled as distinct systems. When these systems are combined in a single corpus, they form a composite Markov chain where transitions between the systems are rare compared to transitions within them.
Experimental Evaluation
The primary evaluation used the novel Lavwa Egal (The Equal Voice) by Thérèse Léotin (2003), which features a mix of French and Martinican Creole. The data consisted of raw text without any prior linguistic annotation or statistical models.
Results and Discussion
The spectral analysis successfully identified generative systems corresponding to the actual languages used in the text. Key insights include:
- Syntagmatic vs. Paradigmatic: Syntagmatic relationships (neighbors) identify different languages, while paradigmatic relationships (shared contexts) identify parts of speech.
- Probabilistic Interpretation: Eigenvector coordinates represent the probability of observing a word given a specific generative system.
- Handling Ambiguity: Words appearing in both languages (e.g., "tout" or "la") gravitate toward the center of the spectral axes, reflecting their shared status.
- Visual Segmentation: By mapping the top three non-trivial eigenvectors to RGB values, the text can be visually segmented, making language boundaries immediately apparent to a human reader.
Conclusion and Future Work
While the system identifies clusters automatically, it currently lacks the ability to label them (e.g., identifying which cluster is "French") without minimal human supervision. Future work aims to create a "virtuous cycle" where syntagmatic and paradigmatic analyses inform each other to improve both language segmentation and part-of-speech tagging simultaneously. This method is specifically useful for related languages mixed in corpora where traditional frequency-based checks often fail.
Summary updated