TL;DR
This paper tackles the challenge of accurately identifying European Portuguese (EP) versus Brazilian Portuguese (BP) in NLP models, which are often biased towards BP due to data availability. They introduce a new cross-domain dataset, PtBrVarId, and demonstrate that transformer-based models, when trained with a novel delexicalization strategy, significantly improve variety identification performance, especially in out-of-domain scenarios.
What Problem Does This Paper Address?
Natural language processing (NLP) models are increasingly being developed for specific language varieties to improve their performance and nuance. However, for languages like Portuguese, which has distinct European and Brazilian varieties, a significant imbalance exists in available online data, heavily favoring Brazilian Portuguese. This bias limits the effectiveness of models trained on this data when applied to European Portuguese texts, hindering their adoption in domains requiring EP-specific language. Existing methods for variety identification often struggle with cross-domain generalization, meaning models trained on one type of text (e.g., news) perform poorly on another (e.g., legal documents). This paper aims to bridge this gap by developing a robust system for Portuguese variety identification that can generalize across different domains.
Methods
The core of this work lies in developing a cross-domain language variety identifier (LVI) for Portuguese. The authors propose a two-step training protocol for transformer-based models (specifically BERTimbau).
- Delexicalization: A key technique employed is delexicalization, where named entities and part-of-speech tags are masked or replaced with generic labels during training. This is done to prevent the model from relying on non-linguistic features (like specific names or places) and instead focus on more subtle linguistic cues that differentiate varieties. The probabilities of replacing these tokens are varied as a hyperparameter.
- Cross-Domain Training Protocol:
- Step 1: Hyperparameter tuning is performed by training the model on a single domain from their new dataset and validating it on all other domains. This helps identify hyperparameters that promote generalization.
- Step 2: The model is then trained on the entire dataset, with undersampling applied to ensure balanced representation across domains and labels.
- Baselines: N-gram models with Naive Bayes classifiers are used as a baseline to compare against the transformer-based approach.
Datasets
- PtBrVarId Corpus: The authors introduce a novel, silver-labeled, multi-domain Portuguese LVI corpus. It's compiled from open-license sources across six domains (Journalistic, Legal, Politics, Web, Social Media, and Literature) and includes over 7 million documents, labeled as either European Portuguese (EP) or Brazilian Portuguese (BP). This corpus was created to address the lack of diverse, cross-domain Portuguese LVI resources.
- DSL-TL Dataset: Used for evaluation, this is a standard benchmark for EP/BP discrimination. Documents labeled as "Both" were excluded to align with the training data.
- FRMT Dataset: Also used for evaluation, this dataset was manually annotated for variety-specific translation systems and adapted for the LVI task.
Key Insights and Ideas
- Delexicalization is Crucial for Cross-Domain LVI: The paper strongly suggests that masking entities and thematic content is vital for models to generalize across domains, especially when texts might share similar entities but differ in variety.
- Transformer Models Outperform N-grams: BERT-based models consistently achieve higher F1 scores than N-gram baselines, demonstrating their superior capacity for capturing nuanced linguistic differences.
- Short Texts Pose a Greater Challenge: The Social Media domain, with its very short texts, presents a significant challenge for variety identification due to limited linguistic cues.
- Silver Labeling Quality Matters: While using silver labels is practical, the authors invested in manual annotation of a subset to ensure quality and understand agreement issues, particularly in domains with linguistic overlap.
Key Results
- Performance Boost from Delexicalization: In the FRMT benchmark, delexicalization improved F1 scores by approximately 13 percentage points for N-grams and 10 for BERT.
- BERT Dominance: The BERT model achieved an F1 score of 84.97% on DSL-TL and 77.25% on FRMT, significantly outperforming the N-gram baseline.
- Delexicalized BERT Performance: The delexicalized BERT model (BERTd) achieved an F1 score of 81.26% on FRMT and 76.19% on DSL-TL, showing substantial gains over its non-delexicalized counterpart in certain scenarios.
- DSL-TL Benchmark Challenges: The DSL-TL benchmark, particularly the "entity bucket," highlights the difficulty of variety identification without delexicalization, where models can misclassify texts due to shared entities. The N-gram model achieved a low F1 score of 46.95% in this specific scenario without delexicalization.
Limitations and Future Work
- Limitations:
- The PtBrVarId corpus is silver-labeled, meaning it relies on heuristics and may contain some noise.
- The study focused on two Portuguese varieties (EP and BP) and did not explore other less-resourced varieties.
- The impact of delexicalizing the validation set was not explored.
- Future Work:
- Expand the corpus to include other Portuguese varieties, especially African Portuguese.
- Investigate the impact of the original training data of pre-trained models (e.g., BERT trained on BP vs. EP) on LVI classifier bias.
- Explore the effect of delexicalizing validation sets.
Summary updated