Analyse spectrale des textes : détection automatique des frontières de langue et de discours

Pascal Vaillant, Richard Nock, Claudia Henry

Introduction

In natural language processing, language identification is often treated as a solved problem for long, monolingual texts. However, "composite" texts—where multiple languages or genres are intertwined—present a significant challenge. Traditional supervised methods like n-gram analysis require extensive labeled datasets for every target language. This research addresses the problem of unsupervised segmentation: identifying boundaries between closely related languages or discourse styles in raw text without prior knowledge or pre-trained models.

Methodology: Spectral Analysis of Markov Chains

The proposed framework utilizes spectral analysis of word transition matrices. The process begins by constructing a "neighborhood matrix" ($W$) based on word co-occurrences, which is then normalized into a transition matrix ($P$). This matrix represents a Markov process where a mixed text is viewed as a combination of several independent "sub-processes" or generative systems.

Markov chain modeling of mixed texts: individual generative systems (left) are combined into a single transition matrix (right).

The core of the method involves the eigenvector decomposition of the transition matrix. While the primary eigenvector captures the global word distribution, subsequent eigenvectors (associated with the largest eigenvalues less than 1) reveal the underlying clusters or "generative systems." This "soft clustering" approach provides a continuous classification, allowing words to belong to multiple systems with varying probabilities—a crucial feature for handling function words shared across languages.

Experimental Evaluation

The method was evaluated on Lavwa Egal — La voix égale, a novel by Thérèse Léotin written in both French and Martinican Creole. These languages are sociolinguistically close and share significant vocabulary, making them an ideal test case for "blind" segmentation. By mapping the coordinates of words in the top three eigenvectors to the RGB color space, the authors can re-display the original text with colors that make language boundaries visually apparent.

Key Insights and Results

  • The Geometry of Language: Words from different systems naturally migrate toward opposite poles along the principal axes of the transition matrix. In the spectral space, language-specific markers (e.g., Creole yo, ka vs. French le, les) clustered distinctly.
  • Syntagmatic vs. Paradigmatic: The focus on words that appear together (syntagmatic) rather than those that share contexts (paradigmatic) allows for the identification of language boundaries rather than just syntactic or semantic classes.
  • Handling Overlap: The system correctly identified that shared vocabulary (like en, y, and an) appeared in different transition contexts, placing them in the "neutral" zone of the spectral space.
  • Probability Interpretation: The authors formally prove that squared eigenvector coordinates, weighted by frequency, represent the probability distributions of words within specific clusters.

Conclusion

Spectral analysis offers a powerful, unsupervised tool for document segmentation. While labeling the resulting clusters still requires minimal human intervention (e.g., identifying "Cluster A" as "French"), the ability to automatically discover generative systems within a corpus opens new avenues for analyzing low-resource or mixed-language data. Future work aims to integrate syntagmatic and paradigmatic analysis to simultaneously learn languages, genres, and syntactic categories from scratch.

Summary updated

Abstract

We propose a theoretical framework within which information on the vocabulary of a given corpus can be inferred on the basis of statistical information gathered on that corpus. Inferences can be made on the categories of the words in the vocabulary, and on their syntactical properties within particular languages. Based on the same statistical data, it is possible to build matrices of syntagmatic similarity (bigram transition matrices) or paradigmatic similarity (probability for any pair of words to share common contexts). When clustered with respect to their syntagmatic similarity, words tend to group into sublanguage vocabularies, and when clustered with respect to their paradigmatic similarity, into syntactic or semantic classes. Experiments have explored the first of these two possibilities. Their results are interpreted in the frame of a Markov chain modelling of the corpus' generative processe(s): we show that the results of a spectral analysis of the transition matrix can be interpreted as probability distributions of words within clusters. This method yields a soft clustering of the vocabulary into sublanguages which contribute to the generation of heterogeneous corpora. As an application, we show how multilingual texts can be visually segmented into linguistically homogeneous segments. Our method is specifically useful in the case of related languages which happened to be mixed in corpora.
BibTeX
@inproceedings{vaillant2006analyse,
  author = {Pascal Vaillant and Richard Nock and Claudia Henry},
  title = {{Analyse spectrale des textes : détection automatique des frontières de langue et de discours}},
  booktitle = {TALN (Posters)},
  year = {2006},
  url = {https://openreview.net/forum?id=hkxYostzoF},
}
Figure