BharatSynthQA: A Context-Grounded Question Answering Dataset on Indian Temples and Religious Heritage

Riya Naik, Shivam Sharma, Swati AgarwalarXiv 2026 · arXiv

TL;DR

BharatSynthQA is a new, large-scale dataset of approximately 99,000 context-grounded question-answering (QA) pairs specifically focused on Indian temples and religious heritage. By combining automated Wikipedia crawling with LLM-based generation and judging, the authors highlight a significant "knowledge gap" in current AI models regarding Indian cultural history, while providing a robust benchmark for retrieval-augmented generation (RAG).

What Problem Does This Paper Address?

While modern Large Language Models (LLMs) are impressively general, they often stumble when queried about niche or culturally specific domains. Indian religious heritage—encompassing intricate architecture, millennia of history, and diverse regional traditions—is one such area where training data is sparse.

Existing QA datasets like SQuAD or TyDi QA are either too broad or lack the specific "grounding" (linking an answer directly to a source passage) necessary for reliable information retrieval. Without this grounding, AI assistants are prone to hallucination. The authors set out to create a dedicated resource that can both evaluate how much LLMs "know" about Indian heritage and serve as a training set for more culturally aware AI.

Methods

The researchers developed a multi-stage pipeline to synthesize high-quality, grounded QA pairs:

  1. Topic-Based Crawling: They started with "seed" documents from authentic sources, such as official temple trust websites and government tourism portals. A crawler then traversed Wikipedia, using cosine similarity of topics (via the TextRazor API) to identify relevant articles.
  2. Geographic Filtering: To ensure the dataset remained focused on Indian heritage, they used OpenStreetMap (OSM) and Google Geocoding to filter out sites located outside of India.
  3. Few-Shot Generation: After collecting ~300 human-annotated QA pairs as examples, they used Llama 4 Scout to generate three types of questions from the crawled passages: Boolean (Yes/No), Factoid (names, dates), and Explanatory (complex reasoning).
  4. LLM-as-a-Judge: To maintain quality, they employed DeepSeek V4 Flash as a judge to score the generated pairs. Only pairs receiving a perfect correctness score (5/5) were retained for the final dataset.
Distribution of topic similarity and the final relevance split of the corpus.

Datasets

The primary contribution is BharatSynthQA, which consists of:

  • 98,677 judge-retained QA pairs (split into training, validation, and test sets).
  • 300 human-annotated pairs released as a high-quality reference set.
  • The source corpus includes 7,617 relevant Wikipedia documents.

The dataset is categorized by question type, with Factoids making up the majority (61.9%), followed by Boolean (20.7%) and Explanatory "Other" questions (17.4%).

The distribution of question types in the human-curated vs. the full generated dataset.

Key Insights and Ideas

  • The Knowledge Gap is Real: Benchmarking showed that LLMs perform poorly on this domain when relying solely on their internal parameters ("closed-book" setting). This suggests that Indian heritage is underrepresented in standard pre-training corpora.
  • Context is the Great Equalizer: When provided with the source passage, model performance skyrocketed. This indicates that the models possess the reasoning capability to understand the content; they simply lack the factual knowledge of the domain.
  • Explanatory Questions are the Frontier: The "Other" category (explanatory questions) proved the most difficult for both the generator and the judge, showing lower retention rates and lower ROUGE-L scores during benchmarking.
  • Efficiency in Judging: The authors found that DeepSeek V4 Flash provided a near-perfect balance of human-agreement (91%) and cost-efficiency ($0.325 per 1,000 pairs), making it a viable alternative to more expensive models like GPT-4 for dataset curation.

Key Results

The researchers benchmarked five models (including DeepSeek, GPT-6 Luna, and Muse Glimmer) under two conditions:

  • Without Context: Factoid token F1 scores were dismal, ranging from 0.18 to 0.30.
  • With Context: Providing the source passage boosted Factoid F1 scores to 0.73–0.77.
  • Boolean Accuracy: Jumped from a baseline of 0.62–0.81 to an impressive 0.93–0.98 when grounded in text.
Fig. 2. Source documents: (a) document length with the median and mean marked; (b) most frequent terms after lemmatization and sized by token count.

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Religious Diversity: The current dataset is heavily "temple-centered" and dominated by Hindu traditions. Future versions aim to broaden the scope to other religions and traditions within India.
  • Language Barrier: The source material is currently limited to English Wikipedia. Expanding to Indian regional languages is a logical next step.
  • Retrieval Complexity: In this study, the "gold" passage was provided to the model. A more realistic "Future Direction" involves testing models in a full RAG setting where they must first find the correct passage among thousands of distractors.

The dataset is publicly available on Hugging Face.

Summary updated

Abstract

India's temples and other places of worship are among its richest repositories of cultural heritage, bringing together religious tradition, history, architecture, and art. Due to the widespread curiosity, people increasingly turn to Artificial Intelligence (AI) assistants for information about them. Current models are rarely trained or evaluated on this domain. We present BharatSynthQA, a passage grounded question answering (QA) dataset focused on Indian temples, which also covers other religious sites connected to them. Our pipeline crawls Wikipedia from topics seeded by authentic heritage sources, filters articles for topical relevance, and locations in India. Further it generates Boolean, factoid, and explanatory QA pairs with a fewshot prompted LLM. An LLM judge filters the pairs on answer correctness, retaining 98,677 of 131,494 generated pairs. Additionally, 300 human-annotated pairs are released separately as a reference set. Human evaluation of the subsample of this dataset showed 89.83% of the questions were formed correctly. Benchmarking five current LLMs shows that factoid token F1 is only 0.18-0.30 without the source passage but 0.73-0.77 with it, indicating that this domain is poorly captured in model knowledge. The dataset is available at https://huggingface.co/datasets/theshivam7/bharatsynth-qa to support research on culturally grounded QA and retrieval-augmented generation for Indian heritage.
BibTeX
@misc{naik2026bharatsynthqa,
  author = {Riya Naik and Shivam Sharma and Swati Agarwal},
  title = {{BharatSynthQA: A Context-Grounded Question Answering Dataset on Indian Temples and Religious Heritage}},
  year = {2026},
  eprint = {2611.11952},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11952},
}
Figure