Disentangling language change: sparse autoencoders quantify the semantic evolution of indigeneity in French

Jacob Matthews, Laurent Dubreuil, Imane Terhmina, Yunci Sun, Matthew Wilkens, Marten van SchijndelNAACL 2025 · DOI

Tracking the evolution of word meanings over centuries—known as diachronic semantic change—is fundamental to historical linguistics. Traditionally, researchers have used Contextual Word Embeddings (CWEs) to measure these shifts. While CWEs capture semantic proximity through dense vectors, they lack interpretability; a change in vector distance signals a shift but fails to explain its nature (e.g., a move from scientific to political usage). Furthermore, historical datasets often suffer from temporal imbalances, complicating traditional clustering.

This study addresses these challenges by applying sparse autoencoders (SAEs) to the French term indigène ("indigenous") between 1825 and 1950. By decomposing dense embeddings into sparse, monosemantic features, the researchers quantify the term's transition from a botanical descriptor to a legal-colonial category.

Methodology

The researchers employed a $k$-sparse autoencoder ($k$-SAE) framework to "unpack" the 1,024-dimensional embeddings generated by CamemBERT-large. The SAE maps these dense vectors into a higher-dimensional sparse space (up to 4,096 dimensions), where each dimension ideally represents a single, human-interpretable concept.

The research pipeline: from sentence extraction and embedding to k-SAE training and automated feature interpretation.

To interpret these features, an automated pipeline was used: GPT-4o (the Interpreter) analyzed sentences with high activation for a specific feature to generate a label, while GPT-4o-mini (the Predictor) validated these labels against new data to ensure consistency.

Data and Temporal Analysis

The study utilized a massive corpus from the French National Library (BNF), spanning 71,000 volumes and 3.9 billion words. From this, 210,305 sentences containing indigène(s) were extracted and divided into five 25-year temporal bins. This granular approach allowed for tracking the activation levels of specific semantic features over time.

Key Findings: From Botany to Colonialism

The SAE successfully disentangled the "superposition" of meanings within the word indigène. In the early 19th century, the term primarily activated botanical and agricultural features (e.g., native plant species). However, after 1875, these activations declined sharply, replaced by features associated with colonial administration and the Code de l’indigénat.

A critical methodological insight was the limitation of standard cosine similarity. While cosine similarity between embeddings across time periods remained high (near 0.99), the Jensen-Shannon (JS) distance of SAE feature activations revealed a clear linear divergence. This demonstrates that SAEs are significantly more sensitive to historical semantic shifts than traditional metrics.

Comparison of distance metrics: Cosine distance on raw embeddings shows minimal change, while JS distance on SAE features reveals significant semantic divergence over time.

Conclusion and Limitations

The model identified 281 highly interpretable features that align closely with historical records, such as the formalization of colonial legal structures. Despite challenges like OCR noise and temporal data imbalance, the use of SAEs provides a powerful tool for "distant reading" of historical archives. Future work aims to apply this framework to other contested terms across different languages to map the global evolution of colonial discourse.

Summary updated

BibTeX
@inproceedings{matthews2025disentangling,
  author = {Jacob Matthews and Laurent Dubreuil and Imane Terhmina and Yunci Sun and Matthew Wilkens and Marten van Schijndel},
  title = {{Disentangling language change: sparse autoencoders quantify the semantic evolution of indigeneity in French}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.559},
}
Figure