Abstract: Not available
Introduction
Traditional search engines, such as the one powering the Australian Web Archive at the National Library of Australia (NLA), have long relied on lexical or "sparse" retrieval methods like BM25 and Solr. While these systems are highly scalable and efficient for massive datasets—including the NLA's 100TB archive of over 2 billion files—they often fail to capture semantic meaning, relying instead on exact keyword matches.
In collaboration with IBM, the NLA explored upgrading this legacy infrastructure by integrating Large Language Models (LLMs) and neural rerankers. This study compares a traditional baseline against two modern architectures: a hybrid "Reranked-NLA" approach and an end-to-end "Neural-NLA" approach. The findings suggest that a hybrid strategy offers the best balance of accuracy, cost-effectiveness, and infrastructure efficiency for large-scale real-world applications.
Methodology
The researchers evaluated two primary upgrade paths against the existing Solr-based baseline:
1. Reranked-NLA (The Hybrid Approach)
This architecture leverages the strengths of both lexical and neural systems. As shown in Figure 1, the process begins with Llama 2 extracting key entities and keywords from a user's natural language query. These keywords are then used to perform a first-stage retrieval from the existing Solr index. The resulting candidate set is re-ordered by a ColBERT neural reranker, which uses "late interaction" to deeply compare query and document tokens. Finally, Llama 2 generates a summary of the top results and suggests follow-up questions.
2. Neural-NLA (The End-to-End Approach)
This pipeline follows a standard dense retrieval model. Queries and documents are converted into embedding vectors using the all-MiniLM-L6-v2 model and stored in a Milvus vector database. Retrieval is performed via vector similarity search.
Quantitative Results
The systems were tested using a pilot set of 49 complex queries where the original system historically underperformed. While pure neural retrieval (Neural-NLA) showed high recall, it struggled with precision and ranking the most relevant answer at the top of the list.
As detailed in Figure 4, the hybrid Reranked-NLA achieved a Mean Reciprocal Rank (MRR) of 0.54, a significant improvement over both the baseline (0.38) and the pure neural approach (0.35). Furthermore, its NDCG@10 score of 0.86 doubled the baseline's performance, indicating a much higher quality of result ranking.
Discussion and Insights
Several key lessons emerged from this demonstration:
- The Domain Shift Challenge: Pure neural retrieval (Neural-NLA) suffered from a "domain shift" because the underlying embedding models were trained on general internet data rather than the specific historical and regional language of the Australian Web Archive. This led to lower precision compared to the lexical baseline.
- LLMs as Metadata Filters: Using Llama 2 for keyword extraction proved to be a highly effective "cleaner" for search queries. This allowed the legacy Solr system to perform high-speed exact matching on refined terms, which the neural reranker then polished for semantic nuance.
- Infrastructure Efficiency: For a 100TB archive, building a full vector index is computationally prohibitive. The Reranked-NLA approach is significantly more efficient as it only applies intensive neural processing to a small subset of documents already filtered by the lexical engine.
- User Preference: Qualitative studies with 14+ participants mirrored the quantitative data. Users rated the summary quality of the hybrid approach at 3.5/5, compared to 2.4/5 for the pure neural approach, suggesting that superior retrieval directly improves the quality of LLM-generated summaries.
Conclusion
The integration of LLMs into large-scale search applications does not require a total replacement of legacy systems. By adopting a hybrid architecture that uses LLMs for query refinement and neural models for reranking, organizations can achieve state-of-the-art semantic search capabilities while maintaining the scalability and cost-efficiency of their existing lexical infrastructure.
Summary updated