Introduction
While Large Language Models (LLMs) dominate the headlines, smaller models like BERT remain the workhorse for many real-world applications—especially when privacy, cost, or hardware constraints are a factor. However, BERT models struggle with "low-resource" languages that lack massive amounts of annotated data. This paper introduces a specialized fine-tuning pipeline for BERT models to boost sentiment analysis performance in languages like Slovak, Maltese, and Icelandic. By integrating Active Learning (AL) with data clustering and dynamic "AL schedulers," the researchers achieved significant savings in annotation costs and notable boosts in F1 scores compared to standard fine-tuning.
Methods
The authors propose a novel pipeline that moves beyond simple fine-tuning by combining several advanced techniques:
- Epistemic Neural Networks (ENNs): Instead of a standard BERT head, the pipeline utilizes an "Epinet." This allows the model to explicitly measure its own "epistemic uncertainty," identifying when it is guessing due to insufficient data exposure.
- Active Learning (AL): The model identifies which unlabeled samples would be most helpful to learn from next. The study evaluated functions such as Entropy, BALD, and Variance sampling.
- Clustering: Agglomerative Hierarchical Clustering was used to group similar text samples. This ensures the model sees a diverse range of data by selecting representative "medoids" or "furthest-batch" samples.
- AL Schedulers: These schedulers modulate the number of samples used over time. For instance, a Linear Scheduler might start by evaluating 50% of the data and gradually decrease that budget to 5% as model confidence grows.
- Recalculating vs. Accumulating: Unlike standard AL which keeps all previously seen data ("Accumulating"), the authors introduced "Recalculating," where the model re-samples its training set from scratch each epoch. This allows the model to "drop" samples that are no longer informative as weights evolve.
Experimental Setup
The study evaluated four languages with varying levels of resource availability:
- Slovak: 30k movie reviews (CSFD dataset).
- Maltese: 851 samples from news and social media (a genuinely low-resource setting).
- Icelandic: 25k samples (translated IMDB reviews).
- Turkish: 25k samples (downsampled tweets to simulate a low-resource environment).
Key Results and Discussion
The proposed pipeline outperformed standard BERT fine-tuning across all tested languages.
Performance Gains
As shown in the table below, the Maltese model (BERTu) saw a significant +4.02 F1 score improvement, while Icelandic (IceBERT) improved by +2.17 points.
Data Efficiency
The models reached baseline performance levels using only a fraction of the data. For Turkish and Maltese, the pipeline achieved baseline results using only ~28% of the available data, demonstrating the power of the active learning approach.
Insights
- The Power of Forgetting: The "Recalculating" strategy proved highly effective. By not being forced to keep every sample from previous epochs, the model could avoid getting stuck on "unlearnable" outliers that cause instability.
- LoRA Limitations: Surprisingly, LoRA (Low-Rank Adaptation) consistently underperformed compared to full fine-tuning in these low-resource scenarios, suggesting that parameter-efficient updates may not be expressive enough for these specific language patterns.
- Stability: The learning curves for the AL-enhanced models were much smoother and more consistent across different random seeds compared to the erratic behavior of the baseline models.
Limitations and Future Work
While the results for Maltese are impressive, the Turkish and Icelandic experiments used downsampled versions of larger datasets. Real-world low-resource environments may have noisier data than these curated sets. Additionally, Active Learning adds computational overhead during the training phase. Future work should explore whether these "scheduling" and "recalculating" strategies translate to larger decoder-style models in similar low-resource settings.
Summary updated