SRS-Stories: Vocabulary-constrained multilingual story generation for language learning

Wiktor Kamzela, Mateusz Lango, Ondrej DusekEMNLP 2025

TL;DR

This paper introduces SRS-Stories, a novel system that generates personalized stories for language learners using only their known vocabulary. By integrating with Spaced Repetition Systems (SRS), it aims to make vocabulary acquisition more engaging and effective than traditional flashcards, offering a more natural way to learn and review words in context.

What Problem Does This Paper Address?

Traditional Spaced Repetition Systems (SRS) like Anki are effective for memorizing vocabulary but often present words out of context, making it difficult for learners to use them naturally in sentences. This can lead to tedious review sessions and limited actual language acquisition. Graded readers offer vocabulary in context, but learners have little control over the vocabulary, and finding appropriate level readers can be challenging, especially for less common languages. This paper addresses the gap by proposing a system that combines the benefits of both: personalized vocabulary control from SRS and contextual learning from graded readers, generated automatically.

Methods

The core of the SRS-Stories system is a Large Language Model (LLM) tasked with generating stories that adhere to specific vocabulary constraints. The authors propose three main strategies for story generation:

  • Simple Prompting: A direct prompt to the LLM to write a story of a certain length, using selected new words multiple times in clear contexts.
  • Planning: A multi-step approach where the LLM first generates story ideas, selects the most promising one, creates an outline, and then generates the final story from the outline. This aims for better coherence and planning.
  • Examples First: The LLM is first prompted to generate example sentences for each target word, which are then used to guide the story generation process, aiming for more natural word integration.

To enforce the vocabulary constraints (ensuring only known or target words are used), they developed three text rewriting strategies:

  • Rewrite (base): The LLM is given a list of unknown words and asked to replace them with simpler alternatives.
  • Rewrite Highlighted: Unknown words are highlighted in the text (e.g., with asterisks), and the LLM is asked to simplify them.
  • Get Synonyms then Rewrite: The LLM first generates synonyms for unknown words and then rewrites the story using these synonyms.

These generation and rewriting strategies can be combined.

Datasets

The experiments were conducted across three languages: English, Polish, and Chinese.

  • Vocabulary Levels:
    • English: Used CEFR-J (Common European Framework of Reference for Languages - Japanese adaptation) word lists.
  • Chinese: Used word lists from the New HSK 3.0 standard.
  • Polish: Word lists were compiled from a frequency list based on the Polish Wikipedia part of the Leipzig Corpora Collection.
  • Language Proficiency Levels: The study evaluated stories at different proficiency levels, corresponding to CEFR A1-C1 for English and HSK 3-5 for Chinese. For Polish, specific CEFR levels were mapped to word counts.

The paper does not explicitly mention specific datasets for LLM training, but rather uses pre-trained LLMs.

Key Insights and Ideas

  • Contextual Learning is Key: The results strongly suggest that learning vocabulary within a story context is significantly more effective and engaging than traditional flashcard-based SRS.
  • LLMs as Storytellers for Learning: LLMs, when properly prompted, can generate coherent, grammatically correct, and interesting stories that effectively introduce and reinforce new vocabulary.
  • Prompting Strategies Matter: The "Planning" and "Examples First" prompting strategies, while more computationally intensive, generally lead to better story quality (coherence, interestingness, grammaticality) compared to "Simple Prompting."
  • Rewriting is Crucial for Constraint Enforcement: The rewriting strategies are essential for ensuring that generated stories adhere to the target vocabulary, with more advanced methods like "Rewrite Highlighted" sometimes improving interestingness but potentially at the cost of vocabulary adherence.
  • Multilingual Potential: The SRS-Stories approach shows promise across multiple languages, though challenges remain in achieving consistent quality, especially for less resourced languages.

Key Results

  • Superiority over Baseline: SRS-Stories, particularly with the "Simple Prompting" and "Rewrite" strategies, significantly outperformed Constrained Beam Search (CBS) in terms of grammaticality, coherence, and interestingness, as assessed by both LLM evaluators and human annotators.
  • Grammaticality and Coherence: The "Examples First" method yielded the highest grammatical correctness, while "Simple Prompting" and "Planning" offered better coherence and interestingness.
  • Vocabulary Inclusion: The "Planning" strategy, combined with "Get Synonyms then Rewrite," was most effective at ensuring new words were used multiple times (#L), with some variants achieving over 2.6 occurrences per word.
  • Out-of-Vocabulary (OOV) Reduction: The rewriting strategies significantly reduced the percentage of out-of-vocabulary words, with some methods achieving OOV rates below 1%.
  • Human Evaluation: Human annotators found SRS-Stories generated by "Simple Prompting" to be significantly more grammatically correct and better at illustrating word usage compared to CBS. Chinese stories, in particular, received very high ratings.
  • LLM vs. Human Agreement: While LLM-based metrics showed moderate correlation with human judgments for English, this correlation was lower for Chinese and Polish, indicating room for improvement in LLM evaluation for multilingual contexts.

Limitations and Future Work

  • Social Biases: The generated stories may reflect social biases present in the LLM's pretraining data, potentially influencing character roles or cultural assumptions.
  • Out-of-Vocabulary (OOV) Words: Despite efforts, some OOV words might still appear, potentially impacting comprehension.
  • Narrative Quality: Narrative quality can vary, sometimes leading to minor inconsistencies or unnatural phrasing.
  • Vocabulary Control: Future work could focus on enhancing vocabulary control and bias mitigation.
  • Initialization of SRS: The paper acknowledges the challenge of initializing SRS with a learner's existing knowledge, which is inherent to SRS-based systems.
  • Phrase/Idiom Handling: While the current experiments focused on single words, the approach can be extended to handle phrases and idioms.
  • Function Words: The paper notes that function words are not ideal for standard SRS but can be intuitively learned through contextual exposure in SRS-Stories.

Summary updated

BibTeX
@inproceedings{kamzela2025srs,
  author = {Wiktor Kamzela and Mateusz Lango and Ondrej Dusek},
  title = {{SRS-Stories: Vocabulary-constrained multilingual story generation for language learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure