Neural Extractive Search

Shauli Ravfogel, Hillel Taub-Tabib, Yoav Goldberg

Abstract: Domain experts often need to extract structured information from large corpora. We advocate for a search paradigm called "extractive search", in which a search query is enriched with capture-slots, to allow for such rapid extraction. Such an extractive search system can be built around syntactic structures, resulting in high-precision, low-recall results. We show how the recall can be improved using neural retrieval and alignment. The goals of this paper are to concisely introduce the extractive-search paradigm; and to demonstrate a prototype neural retrieval system for extractive search and its benefits and potential. Our prototype is available at https://spike.neural-sim.apps.allenai.org/ and a video demonstration is available at https://vimeo.com/559586687.

Introduction

Extracting structured data from massive text corpora typically presents a trade-off between manual document review and resource-intensive Information Extraction (IE) engineering. Traditional search engines are optimized for document retrieval rather than data point extraction. If a researcher requires a list of all risk factors for a specific disease across thousands of papers, they are often left to manually parse search results.

To bridge this gap, we propose "extractive search"—a hybrid paradigm that combines the speed of search with the structured output of IE. By allowing users to define "capture slots" within a search query (e.g., "find all [DRUGS] that treat [DISEASE]"), structured information can be extracted directly from text at scale.

The SPIKE interface for extractive search, showing query graphs and captured entities.

The Extractive Search Paradigm

The foundation of this approach is the SPIKE system, which utilizes symbolic, syntax-based rules to extract information. Users provide a "by-example" query, marking specific spans as slots to be captured. While highly precise, symbolic systems are inherently brittle, failing to capture diverse phrasing or complex grammatical structures. To address this, we augment the symbolic system with neural retrieval and alignment models to improve recall without sacrificing the ease of use provided by the symbolic interface.

Methodology

The Neural Extractive Search pipeline consists of three primary stages designed to expand the reach of symbolic queries:

  1. Symbolic Querying & Encoding: The system initiates a traditional syntactic search to identify a small set of high-precision matches. It then computes the average BERT representations of these matches to create a "dense query vector" representing the semantic relationship.
  2. Neural Retrieval: Utilizing this query vector, the system performs a k-Nearest Neighbor (k-NN) search across a pre-indexed corpus using FAISS. This identifies sentences that are semantically similar to the initial matches, regardless of their syntactic structure.
  3. Neural Alignment: Once similar sentences are retrieved, a span alignment model (based on SciBERT and trained with a triplet loss objective) maps arguments from the source query to the target sentence. This ensures that the correct words are placed into the user's defined capture slots.
The Neural Extractive Search pipeline, combining symbolic search with dense retrieval and neural alignment.

Evaluation and Results

The system was evaluated using the CORD-19 dataset and a curated set of 440 biomedical queries. Key findings include:

  • Recall Expansion: In a case study identifying factors that "spread" COVID-19, neural search increased the number of results from 5 (symbolic) to 40, while simultaneously improving accuracy from 83% to 96%.
  • Relevance and Accuracy: Across diverse biomedical relations, 72.2% of neural results were relevant to the user's intent, and the alignment model correctly identified individual arguments 73.8% of the time.
  • Generalization: Unlike traditional IE models that require training for specific entity types, our alignment model is relation-agnostic and can be applied to new relations at runtime.

Discussion and Future Work

While the prototype demonstrates significant potential, challenges remain regarding latency and the handling of numerical data. Running BERT-based alignment on thousands of candidates is computationally intensive compared to symbolic matching. Future work will focus on scaling these dense retrieval systems and improving performance on quantitative information.

Conclusion

Neural Extractive Search represents a significant step toward unifying search and extraction. By leveraging neural models to provide "fuzzy" matching for symbolic queries, we enable domain experts to perform rapid, structured data extraction from large-scale corpora with minimal engineering overhead.

Summary updated

Abstract

Domain experts often need to extract structured information from large corpora. We advocate for a search paradigm called ``extractive search'', in which a search query is enriched with capture-slots, to allow for such rapid extraction. Such an extractive search system can be built around syntactic structures, resulting in high-precision, low-recall results. We show how the recall can be improved using neural retrieval and alignment. The goals of this paper are to concisely introduce the extractive-search paradigm; and to demonstrate a prototype neural retrieval system for extractive search and its benefits and potential. Our prototype is available at \url{https://spike.neural-sim.apps.allenai.org/} and a video demonstration is available at \url{https://vimeo.com/559586687}.
BibTeX
@inproceedings{ravfogel2021neural,
  author = {Shauli Ravfogel and Hillel Taub-Tabib and Yoav Goldberg},
  title = {{Neural Extractive Search}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-demo.25},
}
Figure