SpiritRAG: A Q&A System for Religion and Spirituality in the United Nations Archive

Yingqiang Gao, Fabian Winiger, Patrick Montjourides, Anastassia Shaitarova, Nianlong Gu, Simon Peng-Keller, Gerold SchneiderEMNLP 2025

TL;DR

This paper introduces SpiritRAG, an interactive Question Answering (Q&A) system designed to help researchers and policymakers navigate the complex and domain-specific information related to religion and spirituality (R/S) within the vast United Nations archive. By leveraging Retrieval-Augmented Generation (RAG), SpiritRAG makes it easier to extract actionable insights from large, often noisy, datasets, addressing a significant gap in digital humanities tools for R/S studies.

What Problem Does This Paper Address?

Religion and spirituality (R/S) are notoriously complex, context-dependent concepts that are difficult to operationalize in traditional archival searches. This challenge is amplified within large, poorly accessible datasets like the UN archive, which are often filled with "noise" from procedural language and irrelevant content. Extracting meaningful R/S-related information requires significant time investment and specialist knowledge. Existing Large Language Models (LLMs) often struggle with the nuanced, domain-specific, and culturally sensitive nature of R/S, leading to potential inaccuracies and a homogenization of perspectives. This paper addresses the need for a specialized system that can effectively retrieve, synthesize, and generate R/S-related knowledge from such challenging information environments.

Methods

SpiritRAG is built upon the Retrieval-Augmented Generation (RAG) paradigm. The system comprises two main components:

  • Document Retrieval: This component uses a pre-trained text encoder (Sentence-BERT and Qwen3-Embedding) to create embeddings for both user queries and the UN resolution documents. It then calculates cosine similarity scores to identify the most relevant documents. A re-ranking step further refines document selection by considering sentence-level similarities between the query and document content.
  • Answer Generation: Instruction-tuned Qwen3 models are used to generate answers. These models take the user's query and the top-k retrieved documents as input, assembling them with a prompt template to produce a coherent and informative response. The system also supports user-uploaded PDF files, which are parsed and incorporated into the generation process for added context. For efficient inference, the VLLM toolkit is employed.

Datasets

  • UN-RES Corpus: A newly constructed corpus of approximately 7,500 UN resolution documents related to health and education, spanning seven languages (Arabic, Chinese, English, French, German, Russian, and Spanish). These documents were collected from the UN Official Document System.
  • Evaluation Test Set: A benchmark of 100 R/S-related questions in health and education, authored by domain experts, used for evaluating the system's performance.

Key Insights and Ideas

  • Bridging the Gap in Digital Humanities for R/S: SpiritRAG demonstrates a practical application of RAG for a highly specialized and under-served domain within digital humanities, showing how advanced NLP can unlock complex archival data.
  • The Challenge of "Actionability": Human evaluations revealed that "actionability" in document retrieval can be low not due to poor retrieval performance, but because UN resolutions themselves may contain limited actionable directives, highlighting the nuanced nature of policy analysis.
  • LLM Limitations in Culturally Sensitive Domains: The paper implicitly underscores the limitations of general-purpose LLMs in handling the cultural and domain-specific nuances of R/S, necessitating specialized systems like SpiritRAG.
  • Importance of Expert Evaluation: The reliance on domain experts for evaluation was crucial in identifying subtle issues, such as the interpretation of "actionability," and in providing qualitative feedback on the system's usefulness.

Key Results

  • Human Evaluation: Pilot tests with domain experts showed promising results. For health-related questions, the Qwen3-Embedding retriever and Qwen3-1.7B generator achieved average ratings of around 4.0-4.9 for Relevance, Accuracy, and Usefulness. For education-related questions, the Qwen3-Embedding retriever and Qwen3-1.7B generator also performed well, with scores generally above 3.5 for Relevance, Accuracy, and Usefulness.
  • System Performance: SpiritRAG is described as lightweight and easily deployable, with an average response time of around 7 seconds for the best-performing setup.
  • Qualitative Feedback: Domain experts perceived SpiritRAG as an innovative and potentially groundbreaking interactive knowledge engine, highlighting its value for future LLM-augmented research in public policy and digital humanities.

Limitations and Future Work

  • Computational Constraints: The evaluation did not include larger-scale LLMs for retrieval or generation due to computational limitations.
  • Lack of Ground Truth: The absence of ground-truth answers and documents meant that the document retriever could not be fine-tuned for optimal performance.
  • Domain Specificity: The current system is tailored to health and education-related R/S questions within UN resolutions.
  • Future Work: The authors plan to expand SpiritRAG to include a broader range of document sources, such as religious texts, to further embed spiritual knowledge. They also aim to extend the system to other UN organizations and multilateral bodies, leveraging its modular architecture.

Summary updated

BibTeX
@inproceedings{gao2025spiritrag,
  author = {Yingqiang Gao and Fabian Winiger and Patrick Montjourides and Anastassia Shaitarova and Nianlong Gu and Simon Peng-Keller and Gerold Schneider},
  title = {{SpiritRAG: A Q\&A System for Religion and Spirituality in the United Nations Archive}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure