LittiChoQA: Literary Texts in Indic Languages Chosen for Question Answering

Aarya Khandelwal, Ritwik Mishra, Rajiv Ratn Shah

Abstract

Long-context question answering (QA) over literary texts poses significant challenges for modern large language models, particularly in low-resource languages. We address the scarcity of long-context QA resources for Indic languages by introducing LittiChoQA, the largest literary QA dataset to date covering many languages spoken in the Gangetic plains of India. The dataset comprises over 270K automatically generated question-answer pairs with a balanced distribution of factoid and non-factoid questions, generated from naturally authored literary texts collected from the open web. We evaluate multiple multilingual LLMs on non-factoid, abstractive QA, under both full-context and context-shortened settings. Results demonstrate a clear trade-off between performance and efficiency: full-context fine-tuning yields the highest token-level and semantic-level scores, while context shortening substantially improves throughput. Among the evaluated models, Krutrim-2 achieves the strongest performance, obtaining a semantic score of 76.1 with full context. While, in shortened context settings it scores 74.9 with answer paragraph selection and 71.4 with vector-based retrieval. Qualitative evaluations further corroborate these findings.

Introduction

Progress in large language model (LLM) question answering has predominantly favored English and high-resource languages. For Indic languages, the research community has often relied on machine-translated versions of English datasets like SQuAD. However, this approach frequently fails to capture the unique linguistic nuances and cultural contexts inherent to the Indian subcontinent.

Furthermore, while "factoid" QA is increasingly handled well by modern models, the real frontier lies in long-context, non-factoid QA. These tasks require a holistic understanding of a narrative to provide descriptive, abstractive answers. LittiChoQA addresses this gap by providing a massive benchmark of naturally authored literary texts from the Gangetic plains, designed to test how well LLMs handle long-form, culturally rich narratives.

Dataset Characteristics

LittiChoQA is the largest literary QA dataset for Indic languages to date, featuring 274,108 QA pairs derived from 3,824 stories. The dataset spans 17 Indic languages, including Hindi, Urdu, Bhojpuri, Maithili, Awadhi, and Braj.

Dataset statistics across 17 Indic languages showing the distribution of stories and QA pairs.

The data generation followed a "synthetic but grounded" approach. Literary texts were gathered from open-web archives and digitized via OCR. QA pairs were then generated directly from the source text using GPT-4.1, ensuring a balanced mix of verbatim factoid questions and inferential non-factoid questions. This native generation avoids the pitfalls of translation-based datasets in low-resource settings.

Methodology

The researchers evaluated six multilingual LLMs, including Llama 3.1, Aya 23, and Krutrim-2, using Parameter-Efficient Fine-Tuning (LoRA). To understand the trade-offs between accuracy and computational cost, three context strategies were compared:

  1. Full Context: The model processes the entire story segment.
  2. Answer Paragraph Selection (APS): A specialized model identifies and selects the most relevant paragraph before answering.
  3. Vector-based Retrieval: A standard Retrieval-Augmented Generation (RAG) approach using sentence embeddings.

Key Results

Experimental results highlight a significant performance-efficiency trade-off. While full-context models provide the most accurate and semantically rich answers, they are computationally intensive. The specialized APS model emerged as a "sweet spot," maintaining high accuracy while drastically increasing throughput.

Performance comparison of various LLMs across full context, APS, and Vector Retriever settings.

Krutrim-2 12B emerged as the top performer, achieving a semantic similarity score (STS MuTe) of 76.1 in the full-context setting. Notably, in shortened context settings, the APS model (74.9) consistently outperformed standard vector-based retrieval (71.4), suggesting that understanding narrative relevance is superior to simple embedding matching for literary tasks. Throughput gains were also massive: shortened-context strategies allowed models to process over 20,000 questions in the same time required for ~900 questions using full context.

Insights and Future Work

The study demonstrates that native models specifically pre-trained on Indic data (like Krutrim-2) can outperform much larger general multilingual models on these specific linguistic tasks. It also underscores that for low-resource languages, generating QA pairs from native texts is a more sustainable path than relying on machine translation.

Future work will aim to address current limitations, such as OCR noise in older digitized manuscripts and the lack of multi-hop reasoning across non-overlapping text chunks. The researchers also plan to expand the dataset to include poetry and explore dynamic context filtering to mitigate the "lost-in-the-middle" problem common in long-document processing.

Summary updated

Abstract

Long-context question answering (QA) over literary texts poses significant challenges for modern large language models, particularly in low-resource languages. We address the scarcity of long-context QA resources for Indic languages by introducing LittiChoQA, the largest literary QA dataset to date covering many languages spoken in the Gangetic plains of India. The dataset comprises over 270K automatically generated question-answer pairs with a balanced distribution of factoid and non-factoid questions, generated from naturally authored literary texts collected from the open web. We evaluate multiple multilingual LLMs on non-factoid, abstractive QA, under both full-context and context-shortened settings. Results demonstrate a clear trade-off between performance and efficiency: full-context fine-tuning yields the highest token-level and semantic-level scores, while context shortening substantially improves throughput. Among the evaluated models, Krutrim-2 achieves the strongest performance, obtaining a semantic score of 76.1 with full context. While, in shortened context settings it scores 74.9 with answer paragraph selection and 71.4 with vector-based retrieval. Qualitative evaluations further corroborate these findings.
BibTeX
@misc{khandelwal2026littichoqa,
  author = {Aarya Khandelwal and Ritwik Mishra and Rajiv Ratn Shah},
  title = {{LittiChoQA: Literary Texts in Indic Languages Chosen for Question Answering}},
  year = {2026},
  eprint = {2601.03025},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.03025},
}
Figure