Introduction
In natural language processing, language identification is often treated as a solved problem for long, monolingual texts. However, "composite" texts—where multiple languages or genres are intertwined—present a significant challenge. Traditional supervised methods like n-gram analysis require extensive labeled datasets for every target language. This research addresses the problem of unsupervised segmentation: identifying boundaries between closely related languages or discourse styles in raw text without prior knowledge or pre-trained models.
Methodology: Spectral Analysis of Markov Chains
The proposed framework utilizes spectral analysis of word transition matrices. The process begins by constructing a "neighborhood matrix" ($W$) based on word co-occurrences, which is then normalized into a transition matrix ($P$). This matrix represents a Markov process where a mixed text is viewed as a combination of several independent "sub-processes" or generative systems.
The core of the method involves the eigenvector decomposition of the transition matrix. While the primary eigenvector captures the global word distribution, subsequent eigenvectors (associated with the largest eigenvalues less than 1) reveal the underlying clusters or "generative systems." This "soft clustering" approach provides a continuous classification, allowing words to belong to multiple systems with varying probabilities—a crucial feature for handling function words shared across languages.
Experimental Evaluation
The method was evaluated on Lavwa Egal — La voix égale, a novel by Thérèse Léotin written in both French and Martinican Creole. These languages are sociolinguistically close and share significant vocabulary, making them an ideal test case for "blind" segmentation. By mapping the coordinates of words in the top three eigenvectors to the RGB color space, the authors can re-display the original text with colors that make language boundaries visually apparent.
Key Insights and Results
- The Geometry of Language: Words from different systems naturally migrate toward opposite poles along the principal axes of the transition matrix. In the spectral space, language-specific markers (e.g., Creole yo, ka vs. French le, les) clustered distinctly.
- Syntagmatic vs. Paradigmatic: The focus on words that appear together (syntagmatic) rather than those that share contexts (paradigmatic) allows for the identification of language boundaries rather than just syntactic or semantic classes.
- Handling Overlap: The system correctly identified that shared vocabulary (like en, y, and an) appeared in different transition contexts, placing them in the "neutral" zone of the spectral space.
- Probability Interpretation: The authors formally prove that squared eigenvector coordinates, weighted by frequency, represent the probability distributions of words within specific clusters.
Conclusion
Spectral analysis offers a powerful, unsupervised tool for document segmentation. While labeling the resulting clusters still requires minimal human intervention (e.g., identifying "Cluster A" as "French"), the ability to automatically discover generative systems within a corpus opens new avenues for analyzing low-resource or mixed-language data. Future work aims to integrate syntagmatic and paradigmatic analysis to simultaneously learn languages, genres, and syntactic categories from scratch.
Summary updated