From Tarzan to Tolkien: Controlling the Language Proficiency Level of LLMs for Content Generation

Ali Malik, Stephen Mayhew, Chris Piech, Klinton Bicknell

Introduction

Large Language Models (LLMs) typically default to a "native" level of proficiency due to their training on massive web-scale datasets. While this is suitable for general tasks, it presents a significant hurdle for specialized applications such as language learning, content creation for children, or simplifying technical documentation. To address this, researchers from Stanford University and Duolingo introduced the Proficiency Control Task (PCT), a framework designed to steer LLMs toward generating text at specific Common European Framework of Reference for Languages (CEFR) levels, ranging from A1 (beginner) to C2 (advanced).

The Challenge of Proficiency Control

The PCT presents two primary difficulties:

  1. Steerability: Standard models often struggle to adhere strictly to a target difficulty level without compromising the narrative quality or relevance of the output.
  2. Evaluation: Measuring "Control Error"—the deviation between a model's output and its target proficiency—requires a formal framework that accounts for content quality and computational overhead.
Distribution of generated CEFR levels for a default model (top) versus the controlled CALM model (bottom), demonstrating the shift from a fixed 'native' level to precise target adherence.

Methodology

The researchers evaluated several strategies to master the PCT:

  • Prompt Engineering: Testing models like GPT-4 and Mistral with varying prompt complexities, from basic instructions to detailed CEFR rubrics.
  • Supervised Finetuning (SFT): Distilling proficiency knowledge from GPT-4 into smaller 7B-parameter models.
  • Reinforcement Learning (PPO): Utilizing Proximal Policy Optimization with an automated CEFR scorer. The negative "Control Error" served as a reward signal to align the model's output with the desired difficulty.
  • Top-$k$ Sampling: An inference-time technique where the model generates multiple candidates, and an external scorer selects the one closest to the target level.

Key Results and the Pareto Frontier

The study culminated in the development of CALM (CEFR-Aligned Language Model). While GPT-4 exhibits strong zero-shot proficiency control, smaller 7B models are initially unresponsive. However, through SFT and PPO, CALM was able to match and eventually exceed GPT-4's accuracy.

The Pareto frontier of Relative Cost (FLOPs) versus Control Error. CALM achieves lower error rates at a fraction of the computational cost of GPT-4.

Key findings include:

  • Superior Accuracy: GPT-4 with complex prompts achieved a Control Error of 0.28. In contrast, CALM combined with top-3 sampling reached a significantly lower error of 0.15.
  • High Quality: Human evaluations for fluency and consistency remained high (4.3–4.7/5), comparable to GPT-4 despite the added constraints.
  • Efficiency: CALM is estimated to be at least 25 times cheaper in terms of FLOPs than GPT-4, making it a viable solution for real-world educational platforms.

Discussion and Limitations

While CALM represents a significant step forward, several challenges remain. The CEFR scale is inherently subjective, leading to a "noisy" ceiling for model performance. Additionally, PPO training proved unstable, requiring careful tuning to avoid model collapse. Future research will focus on extending these techniques to low-resource languages and addressing the cultural biases inherent in the training data.

Summary updated

BibTeX
@inproceedings{malik2024from,
  author = {Ali Malik and Stephen Mayhew and Chris Piech and Klinton Bicknell},
  title = {{From Tarzan to Tolkien: Controlling the Language Proficiency Level of LLMs for Content Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.926},
}
Figure