Elote, Choclo and Mazorca: on the Varieties of Spanish

Cristina España-Bonet, Alberto Barrón-Cedeño

Introduction

Spanish is a global language, serving as the official tongue of 20 countries. However, in the field of Natural Language Processing (NLP), it is frequently treated as a "monolithic" entity. This approach overlooks the profound lexical and cultural diversity inherent in the language. A classic example of this variation is the word for corn: elote in Mexico, choclo in the Andean region, and mazorca in Spain and parts of Colombia.

Map of Spanish linguistic zones and regional terms for corn.

The researchers behind this study argue that ignoring these differences limits the effectiveness of AI models. To address this, they have introduced CEREALex, the largest open-access corpus of Spanish varieties to date, alongside tools designed to identify and analyze regional linguistic nuances.

Building the CEREALex Corpus

The creation of CEREALex involved a multi-stage pipeline. Initially, the team filtered the OSCAR web-crawl dataset, using country-code top-level domains (ccTLDs) like .mx or .ar to provide preliminary labels for documents. This resulted in the CEREAL dataset, containing 41 million documents.

To expand this into CEREALex, the researchers developed a specialized classifier to label the remaining "unknown" portions of the corpus. The final flagship dataset contains 36 million documents and 35 billion words, making it five times larger than previous state-of-the-art resources like the NOW corpus.

Methodology: The docTransformer

A significant challenge in regional classification is that short text snippets often lack the specific lexical markers needed to identify a variety. Standard Transformer models (like BERT) are typically limited to 512 tokens. To overcome this, the authors developed the docTransformer.

Architecture of the docTransformer model for long-document classification.

As shown in the architecture diagram, the docTransformer splits long documents into multiple segments. Each segment is processed through an XLM-RoBERTa (large) model to extract [CLS] token representations. These representations are then aggregated and passed through a classification head (linear layers with tanh activation and dropout). This allows the model to consider the entire document's context before making a final prediction.

Key Insights and Results

The study yielded several critical findings regarding the nature of Spanish varieties in digital spaces:

  • The Importance of Context: The researchers found that classification performance improved significantly when the model could "see" more of the document. Moving from a single 512-token window to two windows (1,024 tokens) resulted in a notable jump in accuracy.
  • High Classification Accuracy: The docTransformer achieved validation accuracies between 94% and 96%. Even on challenging "out-of-domain" test sets (such as news or fashion), it maintained an accuracy of 88-89% across major varieties.
  • The "Mexican" Linguistic Middle Ground: Mexican Spanish was frequently the most common source of misclassification, often being confused with Peninsular Spanish or "Mixed" categories. This suggests it may serve as a dominant or intermediate variety in web-based data.
  • Cultural Mirrors in Embeddings: By generating variety-specific word embeddings, the team demonstrated that these models capture local cultural nuances and social biases more effectively than generic models. For instance, associations between certain concepts varied measurably between Mexican and Peninsular Spanish models.

Limitations and Future Outlook

The authors note that web-sourced data carries an inherent "technological bias," potentially over-representing urban, tech-literate populations. Furthermore, the researchers' own backgrounds in Peninsular and Mexican Spanish influenced the choice of varieties for deep analysis.

Future work aims to explore the "phylogenetic tree" of Spanish varieties by studying embedding space topology. Additionally, the team plans to develop specialized classifiers for under-represented varieties, such as Equatorial Guinean or Philippine Spanish, which currently have minimal digital footprints.

Summary updated

Abstract

Spanish is one of the most widespread languages: the official language in 20 countries and the second most-spoken native language. Its contact with other languages across different regions and the rich regional and cultural diversity has produced varieties which divert from each other, particularly in terms of lexicon. Still, available corpora, and models trained upon them, generally treat Spanish as one monolithic language, which dampers prediction and generation power when dealing with different varieties. To alleviate the situation, we compile and curate datasets in the different varieties of Spanish around the world at an unprecedented scale and create the CEREAL corpus. With such a resource at hand, we perform a stylistic analysis to identify and characterise varietal differences. We implement a classifier specially designed to deal with long documents and identify Spanish varieties (and therefore expand CEREAL further). We produce varietal-specific embeddings, and analyse the cultural differences that they encode. We make data, code and models publicly available.
BibTeX
@inproceedings{espanabonet2024elote,
  author = {Cristina España-Bonet and Alberto Barrón-Cedeño},
  title = {{Elote, Choclo and Mazorca: on the Varieties of Spanish}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.204},
}
Figure