TL;DR
BharatSynthQA is a new, large-scale dataset of approximately 99,000 context-grounded question-answering (QA) pairs specifically focused on Indian temples and religious heritage. By combining automated Wikipedia crawling with LLM-based generation and judging, the authors highlight a significant "knowledge gap" in current AI models regarding Indian cultural history, while providing a robust benchmark for retrieval-augmented generation (RAG).
What Problem Does This Paper Address?
While modern Large Language Models (LLMs) are impressively general, they often stumble when queried about niche or culturally specific domains. Indian religious heritage—encompassing intricate architecture, millennia of history, and diverse regional traditions—is one such area where training data is sparse.
Existing QA datasets like SQuAD or TyDi QA are either too broad or lack the specific "grounding" (linking an answer directly to a source passage) necessary for reliable information retrieval. Without this grounding, AI assistants are prone to hallucination. The authors set out to create a dedicated resource that can both evaluate how much LLMs "know" about Indian heritage and serve as a training set for more culturally aware AI.
Methods
The researchers developed a multi-stage pipeline to synthesize high-quality, grounded QA pairs:
- Topic-Based Crawling: They started with "seed" documents from authentic sources, such as official temple trust websites and government tourism portals. A crawler then traversed Wikipedia, using cosine similarity of topics (via the TextRazor API) to identify relevant articles.
- Geographic Filtering: To ensure the dataset remained focused on Indian heritage, they used OpenStreetMap (OSM) and Google Geocoding to filter out sites located outside of India.
- Few-Shot Generation: After collecting ~300 human-annotated QA pairs as examples, they used Llama 4 Scout to generate three types of questions from the crawled passages: Boolean (Yes/No), Factoid (names, dates), and Explanatory (complex reasoning).
- LLM-as-a-Judge: To maintain quality, they employed DeepSeek V4 Flash as a judge to score the generated pairs. Only pairs receiving a perfect correctness score (5/5) were retained for the final dataset.
Datasets
The primary contribution is BharatSynthQA, which consists of:
- 98,677 judge-retained QA pairs (split into training, validation, and test sets).
- 300 human-annotated pairs released as a high-quality reference set.
- The source corpus includes 7,617 relevant Wikipedia documents.
The dataset is categorized by question type, with Factoids making up the majority (61.9%), followed by Boolean (20.7%) and Explanatory "Other" questions (17.4%).
Key Insights and Ideas
- The Knowledge Gap is Real: Benchmarking showed that LLMs perform poorly on this domain when relying solely on their internal parameters ("closed-book" setting). This suggests that Indian heritage is underrepresented in standard pre-training corpora.
- Context is the Great Equalizer: When provided with the source passage, model performance skyrocketed. This indicates that the models possess the reasoning capability to understand the content; they simply lack the factual knowledge of the domain.
- Explanatory Questions are the Frontier: The "Other" category (explanatory questions) proved the most difficult for both the generator and the judge, showing lower retention rates and lower ROUGE-L scores during benchmarking.
- Efficiency in Judging: The authors found that DeepSeek V4 Flash provided a near-perfect balance of human-agreement (91%) and cost-efficiency ($0.325 per 1,000 pairs), making it a viable alternative to more expensive models like GPT-4 for dataset curation.
Key Results
The researchers benchmarked five models (including DeepSeek, GPT-6 Luna, and Muse Glimmer) under two conditions:
- Without Context: Factoid token F1 scores were dismal, ranging from 0.18 to 0.30.
- With Context: Providing the source passage boosted Factoid F1 scores to 0.73–0.77.
- Boolean Accuracy: Jumped from a baseline of 0.62–0.81 to an impressive 0.93–0.98 when grounded in text.
Limitations and Future Work
The authors acknowledge several areas for improvement:
- Religious Diversity: The current dataset is heavily "temple-centered" and dominated by Hindu traditions. Future versions aim to broaden the scope to other religions and traditions within India.
- Language Barrier: The source material is currently limited to English Wikipedia. Expanding to Indian regional languages is a logical next step.
- Retrieval Complexity: In this study, the "gold" passage was provided to the model. A more realistic "Future Direction" involves testing models in a full RAG setting where they must first find the correct passage among thousands of distractors.
The dataset is publicly available on Hugging Face.
Summary updated