A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus

Eduard Poesina, Cornelia Caragea, Radu Tudor Ionescu

Introduction

Natural Language Inference (NLI) is a fundamental task in Natural Language Understanding (NLU), requiring models to determine the logical relationship—entailment, contradiction, or neutrality—between a premise and a hypothesis. While high-resource languages like English benefit from massive datasets such as SNLI and MNLI, Romanian has remained a low-resource language in this domain. This paper presents RoNLI, the first large-scale Romanian NLI corpus, and introduces a specialized curriculum learning method, Cart-Stra-CL++, designed to handle the noise and class imbalance inherent in automatically generated data.

The RoNLI Dataset

To overcome the prohibitive cost of manual annotation, the authors constructed RoNLI using "distant supervision." By extracting sentence pairs from Romanian Wikipedia linked by specific transition words (e.g., "therefore" for reasoning, "however" for contrast), they generated a training set of 58,000 pairs. To ensure rigorous evaluation, a separate "gold standard" set of 6,000 pairs was manually annotated by native speakers for validation and testing.

Despite its scale, the distant supervision approach introduces two significant challenges: label noise (incorrectly categorized pairs) and severe class imbalance, where "Neutral" and "Reasoning" (Causal) classes vastly outnumber "Contrastive" and "Entailment" samples.

Methodology: Data Cartography and Curriculum Learning

The core innovation of this work is Cart-Stra-CL++, a curriculum learning strategy that leverages Data Cartography. This technique maps training samples based on two metrics calculated during initial training epochs:

  1. Confidence: The model's average probability assigned to the true label.
  2. Variability: The standard deviation of this probability across epochs.

As illustrated in the data cartography plot, samples are categorized into three regions: Easy-to-Learn (E2L) (high confidence, low variability), Ambiguous (A) (mid-range confidence, high variability), and Hard-to-Learn (H2L) (low confidence, low variability).

Data cartography visualization mapping training samples by confidence and variability, identifying easy-to-learn, ambiguous, and hard-to-learn regions.

Standard curriculum learning typically introduces easy samples first. However, the authors observed that in imbalanced datasets, E2L samples are often dominated by majority classes. To counter this, they proposed a Stratified (Stra) approach. Cart-Stra-CL++ ensures that every training batch maintains a diverse representation of all four relationship categories, preventing the model from becoming biased toward majority classes during the early stages of the curriculum.

Experimental Results

The researchers evaluated several architectures, including Ro-BERT (a Romanian-specific BERT) and ChatGPT 3.5. The proposed Ro-BERT + Cart-Stra-CL++ model significantly outperformed baseline methods.

Performance comparison of various models and curriculum learning strategies on the RoNLI dataset.

As detailed in the results table, the stratified curriculum method achieved a Micro F1 score of 0.75 and a Macro F1 of 0.59. Notably, it showed marked improvements in identifying the minority "Contrastive" and "Entailment" classes, where standard models often failed. The study also revealed that "Ambiguous" samples are critical for learning; models trained exclusively on "Easy-to-Learn" data performed poorly, as they lacked the nuanced examples necessary to define complex decision boundaries.

Conclusion

The introduction of RoNLI marks a significant milestone for Romanian NLP. By combining data cartography with stratified curriculum learning, the authors demonstrated that models can effectively learn from noisy, imbalanced, distantly supervised data. The success of Cart-Stra-CL++ on both RoNLI and the English SciNLI dataset suggests that this method is a robust solution for low-resource or noisy NLU tasks across different languages and domains.

Summary updated

BibTeX
@inproceedings{poesina2024novel,
  author = {Eduard Poesina and Cornelia Caragea and Radu Tudor Ionescu},
  title = {{A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.15},
}
Figure