QFrCoLA: a Quebec-French Corpus of Linguistic Acceptability Judgments

David Beauchemin, Richard KhouryEMNLP 2025

TL;DR

This paper introduces QFrCoLA, the first large-scale Quebec-French corpus of linguistic acceptability judgments, designed to evaluate how well language models understand grammatical correctness in this specific dialect. They found that while fine-tuned Transformer models are strong baselines, current large language models struggle with Quebec French acceptability judgments, suggesting a need for more specialized training data and evaluation.

What Problem Does This Paper Address?

Large language models (LLMs) have shown impressive capabilities across various natural language processing (NLP) tasks. However, a significant challenge remains in understanding how these models internalize linguistic knowledge, particularly concerning grammatical correctness and naturalness. To address this, researchers have developed linguistic acceptability judgment benchmarks. While many such benchmarks exist for languages like English, Italian, and Russian, the specific dialect of Quebec French has been underrepresented. This gap makes it difficult to assess how well LLMs perform on this important variant of French, which has distinct linguistic features compared to European French. This paper aims to fill this gap by creating and evaluating a new dataset for Quebec French.

Methods

The authors introduce QFrCoLA (Quebec-French Corpus of Linguistic Acceptability Judgments), a new dataset designed to evaluate LLMs on their ability to distinguish between grammatically correct and incorrect sentences in Quebec French. The dataset consists of 25,153 "in-domain" sentences sourced from the "Banque de dépannage linguistique" (BDL), an official Quebec government resource for linguistic norms, and 2,675 "out-of-domain" (OOD) sentences from the Académie française, a French linguistic authority. Each sentence is labeled as either acceptable (1) or unacceptable (0).

To evaluate LLMs, they employed two main approaches:

  1. Fine-tuned Transformer-based models: They fine-tuned monolingual BERT models for various languages and a cross-lingual XLM-ROBERTa model on the QFrCoLA dataset.
  2. Zero-shot LLM evaluation: They tested eight open-source LLMs (around 7B parameters) in a zero-shot setting, meaning the models were not explicitly trained on the acceptability judgment task but were asked to classify sentences based on their pre-existing knowledge.

Performance was measured using accuracy and the Matthews correlation coefficient (MCC).

Datasets

  • QFrCoLA: The primary contribution, a new dataset of 25,153 in-domain and 2,675 out-of-domain Quebec-French sentences, each labeled for grammatical acceptability.
  • Existing benchmarks: The paper also uses data from other linguistic acceptability corpora for cross-lingual benchmarking, including CoLA (English), DaLAJ (Swedish), ITACOLA (Italian), RuCoLA (Russian), CoLAC (Chinese), NoCOLA (Norwegian), and JCoLA (Japanese).

Key Insights and Ideas

  • Quebec French is distinct: The paper highlights that Quebec French has unique linguistic features (e.g., in title feminization) that differentiate it from European French, necessitating specific evaluation datasets.
  • Normative vs. Speaker-based judgments: QFrCoLA is built from official linguistic resources that reflect norms rather than individual speaker feelings, making it a challenging benchmark for models that might learn from less formal data.
  • LLMs struggle with Quebec French: Despite advancements, current LLMs, even those optimized for French or trained on multilingual data, show limited ability to accurately judge the acceptability of Quebec French sentences.
  • Fine-tuning is crucial: Fine-tuning Transformer models on QFrCoLA significantly improves their performance compared to zero-shot LLMs, underscoring the importance of task-specific and dialect-specific training.
  • Zero-shot LLMs are weak baselines: For this specific task, zero-shot LLMs performed worse than a simple majority baseline, indicating a lack of robust linguistic judgment capabilities in their pre-training for this dialect.

Key Results

  • Fine-tuned BERT on QFrCoLA outperforms SOTA: A fine-tuned BERT model achieved the best performance on the QFrCoLA benchmark, even surpassing the state-of-the-art topological data analysis (LA-TDA) method.
  • Zero-shot LLMs underperform: The tested zero-shot LLMs generally performed poorly, often worse than the naive baseline, suggesting they have not acquired Quebec French linguistic judgment capabilities during pre-training.
  • Cross-lingual models show limited transfer: Pre-trained cross-lingual LLMs did not demonstrate strong linguistic judgment capabilities for Quebec French, even when fine-tuned.
  • OOD performance drop: Models evaluated on the out-of-domain (Académie française) set showed a significant drop in performance, suggesting potential overfitting to the in-domain Quebec French data or highlighting differences between the two French variants.
  • Category imbalance: The "morphology" category constituted nearly 43% of the QFrCoLA dataset, potentially skewing evaluations towards this linguistic phenomenon.

Limitations and Future Work

  • Category imbalance: The dataset's uneven distribution of linguistic phenomena (e.g., overrepresentation of morphology) might lead to an incomplete evaluation of LLM capabilities.
  • Normative bias: The reliance on official linguistic sources means the dataset represents normative grammar, which might not fully capture the diversity of actual language use.
  • Quebec vs. France French differences: The distinctiveness of Quebec French compared to European French poses a challenge, especially for the out-of-domain split.

Future work proposed includes:

  • Expanding the dataset to include minimal pairs for French, creating a novel benchmark for this format.
  • Generating complementary grammatical/ungrammatical sentences for existing ones.
  • Qualitatively exploring the linguistic phenomena errors generated by LLMs.

Summary updated

BibTeX
@inproceedings{beauchemin2025qfrcola,
  author = {David Beauchemin and Richard Khoury},
  title = {{QFrCoLA: a Quebec-French Corpus of Linguistic Acceptability Judgments}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure