TL;DR
Large Language Models (LLMs) can be powerful tools for language learning, but ensuring they adhere to specific linguistic constraints like CEFR A1 level is challenging. This study explores various prompting strategies to guide LLMs in rephrasing texts for beginner English and Italian learners, finding that while improvements are possible, they come at a significant computational cost and don't always guarantee perfect compliance.
What Problem Does This Paper Address?
Learning a new language involves mastering specific grammatical structures and vocabulary. The Common European Framework of Reference for Languages (CEFR) provides a standardized way to describe language proficiency, with A1 being the most basic level. While LLMs can generate vast amounts of text, their ability to consistently produce content that strictly adheres to CEFR A1 constraints for beginner learners is an underexplored area. This is crucial for creating effective, accessible learning materials. The challenge lies in getting LLMs to simplify complex texts without losing essential meaning or introducing new errors, especially when dealing with highly constrained language levels like A1.
Methods
The researchers investigated four prompting strategies to guide LLMs (specifically ChatGPT GPT-4o, GPT-4o-mini, and Llama 3.3-70b) in rephrasing existing texts to meet CEFR A1 level for English and Italian. These strategies varied along two dimensions:
- Stepwise vs. Joint Processing: Whether grammar and vocabulary simplification were performed simultaneously (1-Step) or in separate stages (2-Step).
- Chain-of-Thought (COT): Whether explicit reasoning steps were included in the prompt (COT) or not.
The core of the paraphrasing process involved an iterative approach where the LLM checked the text against CEFR A1 constraints and replaced non-compliant elements. For vocabulary, a simpler prompt encouraged the use of high-frequency words and provided in-text explanations when necessary.
To automatically evaluate the paraphrased texts, the researchers developed a two-part system:
- Lexical Analysis: Using part-of-speech tagging to compare the text's vocabulary against CEFR A1 word lists.
- Grammatical Analysis: Employing GPT-4o with specific prompts to check for adherence to CEFR A1 grammar rules.
A subset of the data was also manually annotated for meaning preservation and linguistic soundness to provide a qualitative assessment.
Datasets
The study utilized sentences collected from Vikidia, an open-access crowd-sourced online encyclopedia for children. This source was chosen for its multilingual content and assumed relative simplicity.
- English Dataset: 1398 original paragraphs.
- Italian Dataset: 737 original paragraphs.
A subset of 100 paragraphs from each language was randomly selected for paraphrasing using the four strategies, resulting in 400 paraphrased texts per language. A further subset of 30 sentences (15 per language) was used for manual annotation.
Key Insights and Ideas
- Guided paraphrasing can improve CEFR A1 compliance, but it's not perfect. LLMs can be steered towards simpler language, but they still struggle to consistently meet all constraints.
- English vs. Italian differences are significant. Strategies that work well for English don't necessarily translate to Italian, and vice-versa. This suggests language-specific nuances in LLM performance and constraint adherence.
- COT is a double-edged sword. While COT strategies showed promise for reducing grammatical violations in English, they significantly increased token usage, making the process less cost-effective.
- Stepwise simplification is better for vocabulary, but joint is better for grammar (in English). For English, combining COT with separate grammar and vocabulary simplification (2-Step COT) emerged as the most effective strategy.
- Italian is trickier. For Italian, one-step strategies seemed better for grammar, while two-step strategies were better for vocabulary, leading to no single "best" strategy. This might be due to Italian's more restrictive grammar rules.
- LLMs still struggle with nuanced linguistic soundness and meaning preservation. Manual evaluations showed low inter-annotator agreement for these aspects, highlighting the difficulty of this task even for human annotators.
- Computational cost is a major factor. COT strategies, while potentially improving quality, dramatically increase token consumption, raising concerns about practicality.
Key Results
- Grammatical Compliance:
- English: Average violations decreased slightly from 1.9 to 1.8, but this was not statistically significant.
- Italian: A substantial reduction in violations was observed, from an average of 4.5 to 2.2, a 52% decrease, which was statistically significant.
- Vocabulary Coverage:
- English: Increased from 42.2% to 57.4% (36% relative improvement).
-
Italian: Increased from 17.6% to 27.6% (57% relative improvement). Both improvements were statistically significant.
-
Strategy Effectiveness: For English, 2-Step COT was found to be the most effective strategy. For Italian, the results were less clear, with different strategies excelling at grammar versus vocabulary.
- Model Performance: GPT-4o models generally showed lower linguistic difference from the original text compared to Llama 3.3-70b. Llama 3.3-70b achieved marginally better vocabulary coverage.
- Token Consumption: COT strategies consistently required significantly more tokens than non-COT strategies, regardless of whether they were 1-Step or 2-Step.
Limitations and Future Work
- Proprietary LLMs: The study relied on proprietary models (OpenAI's GPT-4o and GPT-4o-mini), which may limit reproducibility as API access or model versions change. Future work should explore more open-weight models.
- Manual Annotation Challenges: Low inter-annotator agreement for meaning preservation and linguistic soundness suggests these are difficult aspects to evaluate, even for humans.
- Counterintuitive Correlations: For English, increased grammatical compliance was unexpectedly correlated with reduced A1 vocabulary coverage. This warrants further investigation.
- Cost-Effectiveness: While paraphrasing can improve compliance, the current methods are not cost-effective due to high token usage, especially with COT.
- Future Directions: The authors suggest further investigation into the counterintuitive results, exploring more open-weight models, and potentially developing more efficient prompting strategies. They also highlight the value of their released dataset for future research.
Summary updated