A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus

Eduard Poesina, Cornelia Caragea, Radu Tudor Ionescu

Introduction

Natural Language Inference (NLI) is a fundamental task in natural language understanding, requiring models to determine the logical relationship—entailment, contradiction, or neutrality—between a premise and a hypothesis. While high-quality NLI datasets like SNLI and MNLI exist for English, low-resource languages such as Romanian have lacked dedicated benchmarks. Previous attempts to bridge this gap often relied on machine translation, which frequently fails to capture language-specific nuances and unique syntactic structures.

This paper introduces RoNLI, the first native Romanian NLI corpus. To address the challenges of noise and class imbalance inherent in automatically labeled data, the authors propose Cart-Stra-CL++, a novel stratified curriculum learning method based on data cartography.

Methods: Data Cartography and Cart-Stra-CL++

The authors utilize a multi-stage pipeline to build and train on the RoNLI corpus. Initially, distant supervision is used to scrape Romanian Wikipedia, leveraging specific transition phrases to automatically label 58,000 training pairs.

To manage the noise introduced by automatic labeling, the researchers employ Data Cartography. This technique tracks model behavior during training, mapping samples based on two metrics: confidence (the mean probability assigned to the true label) and variability (the standard deviation of these probabilities).

Data cartography map categorizing samples by confidence and variability into easy, ambiguous, and hard regions.

Samples are categorized into three regions:

  • Easy-to-learn: High confidence, low variability.
  • Ambiguous: High variability, representing the most informative samples for the model.
  • Hard-to-learn: Low confidence, low variability, often indicating mislabeled noise.

The core innovation, Cart-Stra-CL++, addresses the "easy trap" in curriculum learning. Standard curriculum learning (training on easy samples first) can introduce bias if the "easy" samples are dominated by majority classes. Cart-Stra-CL++ uses a stratified approach, ensuring every training batch maintains a representative mix of all classes while gradually increasing the global difficulty level based on a combined confidence-variability score.

The RoNLI Dataset

The primary contribution is the RoNLI corpus, consisting of 64,000 sentence pairs. While the 58,000 training samples are automatically labeled, a subset of 6,000 samples was manually annotated by native speakers for validation and testing to ensure a high-quality benchmark.

Detailed statistics of the RoNLI dataset across different relations and splits.

The dataset covers four relations: Entailment, Contrastive, Causal (Reasoning), and Neutral. Analysis shows that Romanian's syntactic complexity, including its clitic system and preference for subjunctive constructions, makes RoNLI a challenging benchmark for standard transformer models compared to English equivalents.

Key Insights and Results

The study reveals that ambiguous samples are critical for model performance, whereas hard-to-learn samples are typically detrimental noise. By using the Cart-Stra-CL++ method, the researchers achieved significant performance gains.

Benchmarking several models, Ro-BERT (a Romanian-specific BERT) emerged as the top performer. Applying the stratified curriculum learning method improved Ro-BERT’s macro $F_1$ score from 0.56 to 0.59 and its accuracy from 0.73 to 0.75. Furthermore, the method demonstrated strong generalization, outperforming standard training on the English SciNLI dataset with a macro $F_1$ of 0.755.

Conclusion

RoNLI provides a vital resource for Romanian NLP research, moving beyond simple machine translation of English datasets. By combining this new corpus with the Cart-Stra-CL++ training strategy, the authors demonstrate that it is possible to build robust models even from noisy, automatically labeled data. Future work will focus on scaling the manually verified portion of the dataset and exploring architectural adaptations for Romanian's unique linguistic features.

Code and Dataset: https://github.com/Eduard6421/RONLI

Summary updated

BibTeX
@inproceedings{poesina2024novel,
  author = {Eduard Poesina and Cornelia Caragea and Radu Tudor Ionescu},
  title = {{A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.15},
}
Figure