Abstract
Interactive story reading is a common parent-child activity, where parents expect to teach both language skills and real-world knowledge beyond the story. While increasing storytelling and reading systems have been developed for this activity, they often fail to infuse real-world knowledge into the conversation. This limitation can be attributed to the existing question-answering (QA) datasets used for children's education, upon which the systems are built, failing to capture the nuances of how education experts think when conducting interactive story reading activities. To bridge this gap, we design an annotation framework, empowered by existing knowledge graph to capture experts' annotations and thinking process, and leverage this framework to construct StorySparkQA dataset, which comprises 5,868 expert-annotated QA pairs with real-world knowledge. We conduct automated and human expert evaluations across various QA pair generation settings to demonstrate that our StorySparkQA can effectively support models in generating QA pairs that target real-world knowledge beyond story content.
Introduction
Interactive story reading is a cornerstone of early childhood education. When a teacher or parent reads to a child, they often ask questions that bridge the story to the real world—for instance, explaining what a "flood" is when it appears in a narrative. However, current AI-assisted storytelling systems struggle with this "knowledge infusion." Most existing datasets focus on extractive QA, where answers are found directly in the text, leading to simple reading comprehension questions rather than broader conceptual learning.
The challenge lies in the "expert gap": capturing the mental process educators use to identify a concept, link it to external knowledge, and phrase it appropriately for a 3-to-6-year-old. To address this, we introduce StorySparkQA, a dataset of 5,868 expert-annotated QA pairs.
The Expert Annotation Framework
We developed a three-step framework to capture the "thinking process" of 11 education experts:
- Concept Selection: Experts identify educationally suitable words (nouns, verbs, adjectives) from a story section.
- Knowledge Matching: The system retrieves real-world knowledge triples from ConceptNet (e.g.,
[flood, has_subevent, fill]) and definitions from Wiktionary. Experts select the most relevant triple to teach. - QA Pair Creation: Experts write a question and answer based on the selected triple, ensuring it is appropriate for a preschooler's vocabulary and cognitive level.
Dataset Insights
The resulting dataset, StorySparkQA, contains 5,868 QA pairs across 278 classic stories. Analysis of the expert annotations revealed several key insights:
- The "What" Dominance: Experts overwhelmingly preferred "What" questions (86%) over "Why" or "How," aligning with developmental psychology for the 3-6 age group.
- Knowledge Relations: As shown in the distribution of relations, experts frequently used "is a" (35.45%) and "has subevent" (16.21%) to build foundational knowledge.
- Knowledge Graph Utility: Providing experts with ConceptNet triples reduced cognitive load and ensured high-coverage, systematic knowledge infusion.
Experimental Results
We benchmarked several models on the Question-Answer Generation (QAG) task, comparing a fine-tuned T5-Large (770M) against larger LLMs like GPT-4 and Llama 2 (7B).
- Domain-Specific Superiority: The fine-tuned T5-Large achieved the highest Rouge-L score (0.332) and outperformed much larger models in educational appropriateness.
- The GPT-4 Trade-off: While GPT-4 excelled in grammar, it often struggled with "educational appropriateness," using vocabulary that was too advanced for young children.
- Human-Level Quality: Human-annotated QA pairs consistently scored highest in expert evaluations across grammar, relevancy, and consistency.
Conclusion
StorySparkQA demonstrates that for niche educational tasks, domain-specific fine-tuning on expert-annotated data can outperform raw model scale. By bridging the gap between narrative text and real-world knowledge, this dataset paves the way for more engaging and educational AI storytelling systems.
Summary updated