Upgrading Search Applications in the Era of LLMs: A Demonstration with Practical Lessons

Shuang Yu, Nirandika Wanigasekara, Jeff Tan, Kent Fitch, Jaydeep SenIJCAI 2024

Abstract: Not available

Introduction

Traditional search engines, such as the one powering the Australian Web Archive at the National Library of Australia (NLA), have long relied on lexical or "sparse" retrieval methods like BM25 and Solr. While these systems are highly scalable and efficient for massive datasets—including the NLA's 100TB archive of over 2 billion files—they often fail to capture semantic meaning, relying instead on exact keyword matches.

In collaboration with IBM, the NLA explored upgrading this legacy infrastructure by integrating Large Language Models (LLMs) and neural rerankers. This study compares a traditional baseline against two modern architectures: a hybrid "Reranked-NLA" approach and an end-to-end "Neural-NLA" approach. The findings suggest that a hybrid strategy offers the best balance of accuracy, cost-effectiveness, and infrastructure efficiency for large-scale real-world applications.

Methodology

The researchers evaluated two primary upgrade paths against the existing Solr-based baseline:

1. Reranked-NLA (The Hybrid Approach)

This architecture leverages the strengths of both lexical and neural systems. As shown in Figure 1, the process begins with Llama 2 extracting key entities and keywords from a user's natural language query. These keywords are then used to perform a first-stage retrieval from the existing Solr index. The resulting candidate set is re-ordered by a ColBERT neural reranker, which uses "late interaction" to deeply compare query and document tokens. Finally, Llama 2 generates a summary of the top results and suggests follow-up questions.

Architecture of the Reranked-NLA hybrid system, integrating LLM-based keyword extraction and neural reranking.

2. Neural-NLA (The End-to-End Approach)

This pipeline follows a standard dense retrieval model. Queries and documents are converted into embedding vectors using the all-MiniLM-L6-v2 model and stored in a Milvus vector database. Retrieval is performed via vector similarity search.

Quantitative Results

The systems were tested using a pilot set of 49 complex queries where the original system historically underperformed. While pure neural retrieval (Neural-NLA) showed high recall, it struggled with precision and ranking the most relevant answer at the top of the list.

As detailed in Figure 4, the hybrid Reranked-NLA achieved a Mean Reciprocal Rank (MRR) of 0.54, a significant improvement over both the baseline (0.38) and the pure neural approach (0.35). Furthermore, its NDCG@10 score of 0.86 doubled the baseline's performance, indicating a much higher quality of result ranking.

Quantitative performance comparison across baseline, hybrid, and pure neural retrieval models.

Discussion and Insights

Several key lessons emerged from this demonstration:

  • The Domain Shift Challenge: Pure neural retrieval (Neural-NLA) suffered from a "domain shift" because the underlying embedding models were trained on general internet data rather than the specific historical and regional language of the Australian Web Archive. This led to lower precision compared to the lexical baseline.
  • LLMs as Metadata Filters: Using Llama 2 for keyword extraction proved to be a highly effective "cleaner" for search queries. This allowed the legacy Solr system to perform high-speed exact matching on refined terms, which the neural reranker then polished for semantic nuance.
  • Infrastructure Efficiency: For a 100TB archive, building a full vector index is computationally prohibitive. The Reranked-NLA approach is significantly more efficient as it only applies intensive neural processing to a small subset of documents already filtered by the lexical engine.
  • User Preference: Qualitative studies with 14+ participants mirrored the quantitative data. Users rated the summary quality of the hybrid approach at 3.5/5, compared to 2.4/5 for the pure neural approach, suggesting that superior retrieval directly improves the quality of LLM-generated summaries.

Conclusion

The integration of LLMs into large-scale search applications does not require a total replacement of legacy systems. By adopting a hybrid architecture that uses LLMs for query refinement and neural models for reranking, organizations can achieve state-of-the-art semantic search capabilities while maintaining the scalability and cost-efficiency of their existing lexical infrastructure.

Summary updated

BibTeX
@inproceedings{yu2024upgrading,
  author = {Shuang Yu and Nirandika Wanigasekara and Jeff Tan and Kent Fitch and Jaydeep Sen},
  title = {{Upgrading Search Applications in the Era of LLMs: A Demonstration with Practical Lessons}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure