Neural Extractive Search

Shauli Ravfogel, Hillel Taub-Tabib, Yoav Goldberg

Abstract: Traditional search engines excel at document retrieval but struggle with structured fact extraction across large corpora. Neural Extractive Search addresses this by combining symbolic rules with neural similarity and a novel span-alignment model. This hybrid approach allows users to perform high-precision, high-recall information extraction using simple natural language examples. Evaluated on the CORD-19 dataset, the system demonstrates a significant boost in recall (up to 8x) while maintaining high accuracy, transforming complex Information Extraction (IE) tasks into an interactive search experience.

Introduction: The Recall Gap in Information Extraction

Standard search engines are designed to find relevant documents, not to extract structured relationships. For researchers needing to compile comprehensive lists—such as all chemicals used to treat a specific virus—traditional Information Extraction (IE) requires building custom, labor-intensive pipelines for every new relation.

The interactive search interface for extracting structured facts.

Symbolic systems like SPIKE introduced "capture slots" to allow users to define relations (e.g., [Something] is a risk factor for [Stroke]). However, these systems are linguistically brittle; they fail to capture variations like "Stroke is often preceded by X." This "recall gap" limits the utility of symbolic search for comprehensive data gathering. Neural Extractive Search aims to bridge this gap by finding diverse linguistic expressions of the same underlying idea without losing the precision of structured extraction.

Methods: A Hybrid Pipeline

The authors propose a two-stage pipeline that augments symbolic search with neural retrieval and alignment.

Overview of the Neural Extractive Search pipeline.
  1. Symbolic-to-Neural Querying: The process begins with a "by-example" query where the user marks arguments in a sample sentence. The system runs an initial symbolic search to find high-precision matches. It then averages the BERT embeddings of these matches to create a "dense query vector" that represents the semantic essence of the relation, filtering out idiosyncratic noise from any single sentence.
  2. Neural Retrieval: This vector is used for k-Nearest Neighbor (k-NN) search over a pre-indexed corpus using SciBERT and FAISS. This step identifies sentences that are semantically similar to the query but may use entirely different phrasing.
  3. Argument Identification via Alignment: To extract structured data from these neurally retrieved sentences, the system must identify which spans correspond to the user's original capture slots. The authors developed a Neural Alignment Model trained with a triplet loss objective. This model ensures that "correct" spans in a result sentence are mapped closer to the query arguments in vector space than "distractor" spans.

Key Insights

A major contribution of this work is the General-Purpose Alignment model. Unlike traditional IE systems that require retraining for every new relation, this alignment model is trained once on a diverse set of relations (using the SPIKE-Queries Dataset) and generalizes to unseen queries at test time. This enables "Interactive IE," where users can iteratively build structured tables from massive text corpora without needing a dedicated data science team.

Key Results

The system was evaluated on the CORD-19 dataset, containing millions of scientific sentences. Key findings include:

  • Massive Recall Boost: For specific relations like "spreads by," the neural system increased the number of captured facts by 8x (from 5 to 40) with 96% accuracy.
  • High Relevancy: Approximately 72.2% of neurally retrieved results were manually verified as relevant across various biomedical relations.
  • Alignment Precision: The neural alignment model correctly identified capture slot boundaries 73.8% of the time, demonstrating its ability to handle the inherent "fuzziness" of natural language.

Limitations and Future Work

Despite its success, the system faces challenges. Neural alignment is computationally expensive compared to symbolic search, leading to higher latency. Additionally, the model struggles with numerical sensitivity, often retrieving structurally similar sentences with unrelated numerical values. Future research will focus on a "meta-model" to predict which queries are best suited for neural versus symbolic processing to optimize both speed and quality.

Summary updated

Abstract

Domain experts often need to extract structured information from large corpora. We advocate for a search paradigm called ``extractive search'', in which a search query is enriched with capture-slots, to allow for such rapid extraction. Such an extractive search system can be built around syntactic structures, resulting in high-precision, low-recall results. We show how the recall can be improved using neural retrieval and alignment. The goals of this paper are to concisely introduce the extractive-search paradigm; and to demonstrate a prototype neural retrieval system for extractive search and its benefits and potential. Our prototype is available at \url{https://spike.neural-sim.apps.allenai.org/} and a video demonstration is available at \url{https://vimeo.com/559586687}.
BibTeX
@inproceedings{ravfogel2021neural,
  author = {Shauli Ravfogel and Hillel Taub-Tabib and Yoav Goldberg},
  title = {{Neural Extractive Search}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-demo.25},
}
Figure