TL;DR
The SadaqahStories project introduces a verified, parallel English-Arabic corpus of 41 children's stories about Islamic charity. By releasing the original AI-generated drafts alongside audited and final versions, the authors provide a unique dataset for studying Large Language Model (LLM) hallucinations in sensitive religious and historical contexts.
What Problem Does This Paper Address?
Generative AI makes content creation cheap, but it often prioritizes "sounding right" over being factually accurate. In the domain of religious education, where accuracy is paramount, LLMs frequently "hallucinate" details—inventing hadith wordings, historical dates, or specific numbers (like the exact count of camels in a relief effort) that have no basis in source texts.
The authors address the scarcity of high-quality, bilingual (English-Arabic) digital material for children on Islamic giving (Sadaqah, Zakat, and Waqf). More importantly, they tackle the challenge of auditing AI-generated narratives to ensure they are safe and accurate for educational use.
Methods
The researchers followed a rigorous five-stage pipeline to build the corpus:
- Drafting (V0): 62 bilingual story drafts were generated using LLM assistance.
- Selection: Filtering out near-duplicates and stories that lacked verifiable sources or might be religiously contested.
- Audit (V1): A line-by-line check against canonical sources (Qur'an, hadith collections like sunnah.com, and historical records). Every correction was logged and categorized.
- Child-oriented Rewrite (V2): Simplifying the audited text for readers aged 9–12 without adding new facts.
- Scholarly Sign-off: A final review by an Islamic scholar to ensure religious and linguistic integrity.
The team also developed a specialized correction log using a custom error typology to track exactly how the AI-drafted narratives deviated from the truth.
Datasets
The primary output is the SadaqahStories Corpus, which includes:
- 41 stories (328 page-level passages) across three versions: AI-assisted draft (V0), audited (V1), and final (V2).
- A per-story correction log with error codes.
- 58 source links to canonical religious and historical texts.
- Charity Lexicon: An 81-entry dataset of charitable terms in Modern Standard Arabic and Qatari Arabic, normalized for NLP tasks.
Key Insights and Ideas
- Vividness vs. Veracity: LLMs often hallucinate specific, concrete details (prices, names, staff counts) to make a story more engaging. While this works for fiction, it is dangerous for historical and religious narratives.
- Scripture vs. Narrative: While LLMs sometimes misquote scripture, the majority of errors occurred in the historical frame surrounding the scripture—the "who, when, and how much" of the story.
- Language-Specific Hallucinations: The Arabic drafts were not simple translations of the English ones and contained their own unique errors. For example, one Arabic draft incorrectly labeled voluntary charity as Zakat (obligatory alms), a distinction the English version got right.
- The Cost of Correction: Auditing and correcting AI drafts often involves significant "deletion." The audited versions were 38–48% shorter than the AI drafts because so much unsupported "fluff" had to be removed.
Key Results
- High Error Rate: 39 out of the 41 published drafts (95%) contained at least one substantive error.
- Error Typology: The most frequent errors were "Unsupported specific details" (16 stories) and "Fabricated entities/quotations" (9 stories).
- Readability Improvement: The rewriting process successfully lowered the English Flesch-Kincaid grade level from a difficult 14.0 (college level) to a child-friendly 6.1.
- Platform: The corpus is served via a free, account-free website designed to protect minor privacy while providing interactive educational tools.
Limitations and Future Work
The authors acknowledge that the corpus is relatively small (41 stories). The error coding was performed by a single coder, meaning inter-annotator agreement wasn't measured. Additionally, while English readability was measured using standard formulas, a specialized Arabic readability model (like BAREC) is needed for future iterations.
Future work includes measuring how children actually interact with these stories in classroom settings and expanding the corpus by auditing the remaining drafts that didn't make the initial cut.
Summary updated