Introduction
Spanish is a global language, serving as the official tongue of 20 countries. However, in the field of Natural Language Processing (NLP), it is frequently treated as a "monolithic" entity. This approach overlooks the profound lexical and cultural diversity inherent in the language. A classic example of this variation is the word for corn: elote in Mexico, choclo in the Andean region, and mazorca in Spain and parts of Colombia.
The researchers behind this study argue that ignoring these differences limits the effectiveness of AI models. To address this, they have introduced CEREALex, the largest open-access corpus of Spanish varieties to date, alongside tools designed to identify and analyze regional linguistic nuances.
Building the CEREALex Corpus
The creation of CEREALex involved a multi-stage pipeline. Initially, the team filtered the OSCAR web-crawl dataset, using country-code top-level domains (ccTLDs) like .mx or .ar to provide preliminary labels for documents. This resulted in the CEREAL dataset, containing 41 million documents.
To expand this into CEREALex, the researchers developed a specialized classifier to label the remaining "unknown" portions of the corpus. The final flagship dataset contains 36 million documents and 35 billion words, making it five times larger than previous state-of-the-art resources like the NOW corpus.
Methodology: The docTransformer
A significant challenge in regional classification is that short text snippets often lack the specific lexical markers needed to identify a variety. Standard Transformer models (like BERT) are typically limited to 512 tokens. To overcome this, the authors developed the docTransformer.
As shown in the architecture diagram, the docTransformer splits long documents into multiple segments. Each segment is processed through an XLM-RoBERTa (large) model to extract [CLS] token representations. These representations are then aggregated and passed through a classification head (linear layers with tanh activation and dropout). This allows the model to consider the entire document's context before making a final prediction.
Key Insights and Results
The study yielded several critical findings regarding the nature of Spanish varieties in digital spaces:
- The Importance of Context: The researchers found that classification performance improved significantly when the model could "see" more of the document. Moving from a single 512-token window to two windows (1,024 tokens) resulted in a notable jump in accuracy.
- High Classification Accuracy: The
docTransformerachieved validation accuracies between 94% and 96%. Even on challenging "out-of-domain" test sets (such as news or fashion), it maintained an accuracy of 88-89% across major varieties. - The "Mexican" Linguistic Middle Ground: Mexican Spanish was frequently the most common source of misclassification, often being confused with Peninsular Spanish or "Mixed" categories. This suggests it may serve as a dominant or intermediate variety in web-based data.
- Cultural Mirrors in Embeddings: By generating variety-specific word embeddings, the team demonstrated that these models capture local cultural nuances and social biases more effectively than generic models. For instance, associations between certain concepts varied measurably between Mexican and Peninsular Spanish models.
Limitations and Future Outlook
The authors note that web-sourced data carries an inherent "technological bias," potentially over-representing urban, tech-literate populations. Furthermore, the researchers' own backgrounds in Peninsular and Mexican Spanish influenced the choice of varieties for deep analysis.
Future work aims to explore the "phylogenetic tree" of Spanish varieties by studying embedding space topology. Additionally, the team plans to develop specialized classifiers for under-represented varieties, such as Equatorial Guinean or Philippine Spanish, which currently have minimal digital footprints.
Summary updated