A Hybrid Approach to Information Retrieval and Answer Generation for Regulatory Texts

Jhon Stewar Rayo Mosquera, Carlos Raul De La Rosa Peredo, Mario Garrido CordobaCOLING 2025

TL;DR

This paper introduces a hybrid system that combines traditional keyword search (BM25) with advanced semantic search (fine-tuned Sentence Transformers) to improve information retrieval from complex regulatory documents. This hybrid approach, coupled with a Retrieval Augmented Generation (RAG) framework, significantly outperforms standalone methods in finding relevant information and generating accurate, context-aware answers to compliance-related questions.

What Problem Does This Paper Address?

Regulatory texts are notoriously dense, jargon-filled, and complex, making it incredibly difficult for humans and traditional information retrieval (IR) systems to efficiently find the specific information needed for compliance tasks. Standard IR methods, like BM25, rely on exact keyword matching, which often fails when dealing with synonyms, paraphrasing, or domain-specific terminology common in legal and regulatory documents. While semantic search methods using embeddings can capture meaning, they can sometimes miss crucial keyword matches. This paper aims to bridge this gap by creating a more robust IR system that leverages both lexical and semantic understanding, and then uses this enhanced retrieval to power a question-answering system that can generate accurate responses.

Methods

The core of the paper is a hybrid information retrieval system. This system combines two main components:

  • BM25 (Baseline): A well-established algorithm that ranks documents based on term frequency and inverse document frequency. This provides strong lexical coverage.
  • Semantic Retriever: This component uses a fine-tuned Sentence Transformer model (specifically, BAAI/bge-small-en-v1.5). The model was fine-tuned on the regulatory dataset to better understand the semantic nuances of the domain. This allows it to find documents that are semantically similar to the query, even if they don't share exact keywords.

These two retrieval methods are combined using a weighted score (Equation 1), where the semantic score is given slightly more weight (a = 0.65).

For answer generation, the system employs a Retrieval Augmented Generation (RAG) framework. This means:

  1. The hybrid IR system retrieves the top-k relevant passages (up to 10).
  2. These passages are then fed into a Large Language Model (LLM) to synthesize a comprehensive and accurate answer.
  3. The system uses a specific prompt designed to guide the LLM to focus on accuracy, completeness, and alignment with the provided regulatory text, referencing specific obligations and requirements.

A data processing pipeline was also developed, including steps like expanding contractions, normalization, preserving legal formats, removing stopwords, stemming, and tokenization, to prepare the text for retrieval.

Datasets

The primary dataset used is ObliQA, which consists of 27,869 regulatory questions. This dataset was extracted from 40 documents provided by Abu Dhabi Global Markets, an authority overseeing financial services within the European Economic Area. The dataset is divided into training, testing, and validation sets. Each question is paired with one or more passages containing the relevant information.

Key Insights and Ideas

  • Hybrid is better: The paper strongly suggests that combining lexical (BM25) and semantic (fine-tuned Sentence Transformer) retrieval is crucial for effectively navigating the complexities of regulatory texts, where both precise terminology and underlying meaning are important.
  • Domain-specific fine-tuning matters: Fine-tuning a general-purpose language model on regulatory data significantly improves its ability to capture semantic relationships within that specific domain.
  • RAG for compliance: The RAG framework, powered by the hybrid retriever, is effective in generating contextually relevant and accurate answers to regulatory questions, going beyond simple passage retrieval.
  • Prompt engineering is key for LLMs: A carefully crafted prompt is essential for guiding LLMs to produce high-quality, compliant answers that adhere strictly to the provided retrieved information.

Key Results

  • Information Retrieval: The hybrid system significantly outperformed standalone lexical (BM25) and semantic approaches. Notable improvements were seen in Recall@10 and MAP@10. Table 2 shows the hybrid system achieving a Recall@10 of 0.8333 and MAP@10 of 0.7016, compared to BM25 (0.7791, 0.6415) and the semantic system (0.8103, 0.6286).
  • Answer Generation: When evaluated using the Regulatory Passage Answer Stability Score (RePASs), the hybrid retriever with GPT 3.5 Turbo achieved the highest RePASs score of 0.57. This outperformed GPT-4o Mini (0.44) and Llama 3.1 (0.37). However, the paper notes that while the hybrid system shows improvements in contradiction handling and obligation coverage, the entailment score (E5) was lower, indicating room for improvement in how well generated sentences are supported by retrieved passages.

Limitations and Future Work

  • Entailment Score: The authors acknowledge that the entailment score (E5) in the RePASs evaluation was lower than desired, suggesting that the LLM synthesis process could be further optimized to ensure generated answers are more directly and fully supported by the retrieved passages.
  • LLM Fine-tuning: Future work could involve fine-tuning LLMs on domain-specific corpora to enhance their alignment with regulatory contexts.
  • Retrieval Thresholds: Optimizing retrieval thresholds could lead to better semantic coverage.
  • Scoring Mechanisms: Exploring advanced scoring mechanisms to better balance precision and recall in the retrieval phase is also suggested.
  • Model Availability: The authors have made their fine-tuned model and methodology available on Hugging Face Hub and GitHub, encouraging further research and development.

Summary updated

BibTeX
@inproceedings{mosquera2025hybrid,
  author = {Jhon Stewar Rayo Mosquera and Carlos Raul De La Rosa Peredo and Mario Garrido Cordoba},
  title = {{A Hybrid Approach to Information Retrieval and Answer Generation for Regulatory Texts}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2025},
}
Figure