TL;DR
This paper introduces "Historical Ink," a novel pipeline for detecting semantic shifts in 19th-century Spanish texts, particularly focusing on Latin American Spanish. By leveraging fine-tuned BERT models and a custom corpus, the research offers valuable insights into how societal and cultural changes are reflected in language evolution.
What Problem Does This Paper Address?
Understanding how word meanings evolve over time is crucial for historical linguistics and natural language processing. This evolution, known as Semantic Shift Detection (SSD), is challenging because word meanings can change due to social, historical, and political factors. While traditional methods are manual and time-consuming, recent advancements in computational linguistics, particularly with contextual embeddings like BERT, have revolutionized this field. However, much of this research has focused on English, leaving a gap in the systematic analysis of historical Spanish, especially the distinct linguistic landscape of Latin America. This paper aims to bridge that gap by developing and applying a robust SSD pipeline to 19th-century Spanish texts.
Methods
The core of this work is the "Historical Ink SSD Pipeline Architecture," a modular and configurable system. The pipeline involves several key steps:
- Occurrence Finding: Identifying all instances of a target word in both old (Cold) and new (Cnew) corpora. This step is made more robust by handling Spanish's inflectional variations and historical orthographic changes (e.g., "luzes" vs. "luces").
- Word Embeddings: Using fine-tuned BERT-like language models (specifically BETO and ALBERT) to generate contextualized word embeddings. The models were fine-tuned on the specific corpora (19th-century Spanish and its Latin American portion) using Masked Language Modeling (MLM) to capture corpus-specific linguistic styles.
- Clustering: Applying a joint clustering approach to group word embeddings by meaning (sense) across both corpora. This step utilizes algorithms like Affinity Propagation and KMeans, aiming to identify distinct senses and their evolution. The Diachronic Word Usage Graph (DWUG) is generated to visualize these shifts.
- Semantic Shift Measurement: Quantifying the semantic change using metrics like Cosine Distance (CD) and Inverted Similarity over Word Prototype (PRT). This allows for the identification of senses that have been gained or lost over time.
Datasets
- Cold Corpus: A newly constructed 19th-century Spanish corpus (1800-1914) compiled from Project Gutenberg, The British Library books, and the LatamXIX dataset. This corpus was cleaned and chunked to be compatible with BERT models.
- Cnew Corpus: The Spanish portion of the "EUBookShop" corpus (Cañete, 2019) is used as the modern corpus for comparison.
- LSCDiscovery Shared Task Data: Used for evaluating and selecting the best-performing language models. This dataset, with over 65,000 annotated examples for 100 target words, provided a benchmark for assessing model robustness.
Key Insights and Ideas
- Latin American Spanish Distinctiveness: The study highlights the importance of analyzing Latin American Spanish separately, as its linguistic evolution may differ from general Spanish due to unique historical and cultural influences.
- BERT's Efficacy for Historical Texts: Fine-tuned BERT models, particularly BETO cased, demonstrate strong performance in capturing nuanced semantic shifts in historical Spanish, outperforming simpler methods.
- Cultural Shifts Reflected in Language: The analysis of words like "mujeres" (women) and "infancia" (infancy) reveals how societal changes (e.g., gender discourse, nation-building) directly influence word meanings and usage.
- Challenges in Historical Data: The paper underscores the difficulties of working with historical texts, including orthographic variations, inflectional complexity, and the need for careful data cleaning and preprocessing.
- Modularity is Key: The proposed pipeline is designed to be flexible and reusable, allowing for adaptation to different languages, time periods, and research questions.
Key Results
- Model Selection: BETO fine-tuned on the Latin American dataset (BETO cased FT) was identified as the best-performing model for the SSD task, likely due to the additional OCR correction applied to this portion of the corpus and BETO's single-language focus.
- Semantic Shifts Observed:
- "Mujeres" (Women): Shifted from a specific designation for a group of females to a more general term encompassing all women, reflecting evolving gender discourse and the move towards a relational concept of gender.
- "Infancia" (Infancy): Evolved from an abstract reference to the nascent phase of entities or people (including a metaphorical use for newly independent nations) to a modern understanding primarily referring to the population segment of children.
- "Sentimiento" (Sentiment): Lost an older meaning related to morality, artistic spirit, or elevated religious feeling, now primarily referring to personal feelings.
- "Sublime": Has largely fallen out of common usage, with its older meanings no longer prevalent.
- "Servidores" (Servants/Servers): Gained a new sense related to computer hardware and networks.
- Clustering Performance: Affinity Propagation (AP) struggled with multi-meaning words but was effective for single-sense words, while KMeans performed better on multi-meaning words.
Limitations and Future Work
- Limitations:
- The study acknowledges that the LSCDiscovery task differs from their specific focus, meaning direct score comparisons are not applicable.
- The evaluation of models for the Latin American corpus, particularly for clustering, could be more extensive.
- Some words were found to be "wrongly clustered" by AP (e.g., "grave," "honor"), indicating areas for improvement in clustering algorithms.
- Future Work:
- The authors suggest the need for an annotated dataset specifically for Latin American Spanish to facilitate more focused model evaluation.
- The flexible pipeline can be adapted for other languages and historical periods, serving as a framework for future research.
- Further investigation into the specific nuances of word semantics in Latin American Spanish is recommended.
Summary updated