Tracking the evolution of word meanings over centuries—known as diachronic semantic change—is fundamental to historical linguistics. Traditionally, researchers have used Contextual Word Embeddings (CWEs) to measure these shifts. While CWEs capture semantic proximity through dense vectors, they lack interpretability; a change in vector distance signals a shift but fails to explain its nature (e.g., a move from scientific to political usage). Furthermore, historical datasets often suffer from temporal imbalances, complicating traditional clustering.
This study addresses these challenges by applying sparse autoencoders (SAEs) to the French term indigène ("indigenous") between 1825 and 1950. By decomposing dense embeddings into sparse, monosemantic features, the researchers quantify the term's transition from a botanical descriptor to a legal-colonial category.
Methodology
The researchers employed a $k$-sparse autoencoder ($k$-SAE) framework to "unpack" the 1,024-dimensional embeddings generated by CamemBERT-large. The SAE maps these dense vectors into a higher-dimensional sparse space (up to 4,096 dimensions), where each dimension ideally represents a single, human-interpretable concept.
To interpret these features, an automated pipeline was used: GPT-4o (the Interpreter) analyzed sentences with high activation for a specific feature to generate a label, while GPT-4o-mini (the Predictor) validated these labels against new data to ensure consistency.
Data and Temporal Analysis
The study utilized a massive corpus from the French National Library (BNF), spanning 71,000 volumes and 3.9 billion words. From this, 210,305 sentences containing indigène(s) were extracted and divided into five 25-year temporal bins. This granular approach allowed for tracking the activation levels of specific semantic features over time.
Key Findings: From Botany to Colonialism
The SAE successfully disentangled the "superposition" of meanings within the word indigène. In the early 19th century, the term primarily activated botanical and agricultural features (e.g., native plant species). However, after 1875, these activations declined sharply, replaced by features associated with colonial administration and the Code de l’indigénat.
A critical methodological insight was the limitation of standard cosine similarity. While cosine similarity between embeddings across time periods remained high (near 0.99), the Jensen-Shannon (JS) distance of SAE feature activations revealed a clear linear divergence. This demonstrates that SAEs are significantly more sensitive to historical semantic shifts than traditional metrics.
Conclusion and Limitations
The model identified 281 highly interpretable features that align closely with historical records, such as the formalization of colonial legal structures. Despite challenges like OCR noise and temporal data imbalance, the use of SAEs provides a powerful tool for "distant reading" of historical archives. Future work aims to apply this framework to other contested terms across different languages to map the global evolution of colonial discourse.
Summary updated