Abstract: Traditional search engines excel at document retrieval but struggle with structured fact extraction across large corpora. Neural Extractive Search addresses this by combining symbolic rules with neural similarity and a novel span-alignment model. This hybrid approach allows users to perform high-precision, high-recall information extraction using simple natural language examples. Evaluated on the CORD-19 dataset, the system demonstrates a significant boost in recall (up to 8x) while maintaining high accuracy, transforming complex Information Extraction (IE) tasks into an interactive search experience.
Introduction: The Recall Gap in Information Extraction
Standard search engines are designed to find relevant documents, not to extract structured relationships. For researchers needing to compile comprehensive lists—such as all chemicals used to treat a specific virus—traditional Information Extraction (IE) requires building custom, labor-intensive pipelines for every new relation.
Symbolic systems like SPIKE introduced "capture slots" to allow users to define relations (e.g., [Something] is a risk factor for [Stroke]). However, these systems are linguistically brittle; they fail to capture variations like "Stroke is often preceded by X." This "recall gap" limits the utility of symbolic search for comprehensive data gathering. Neural Extractive Search aims to bridge this gap by finding diverse linguistic expressions of the same underlying idea without losing the precision of structured extraction.
Methods: A Hybrid Pipeline
The authors propose a two-stage pipeline that augments symbolic search with neural retrieval and alignment.
- Symbolic-to-Neural Querying: The process begins with a "by-example" query where the user marks arguments in a sample sentence. The system runs an initial symbolic search to find high-precision matches. It then averages the BERT embeddings of these matches to create a "dense query vector" that represents the semantic essence of the relation, filtering out idiosyncratic noise from any single sentence.
- Neural Retrieval: This vector is used for k-Nearest Neighbor (k-NN) search over a pre-indexed corpus using SciBERT and FAISS. This step identifies sentences that are semantically similar to the query but may use entirely different phrasing.
- Argument Identification via Alignment: To extract structured data from these neurally retrieved sentences, the system must identify which spans correspond to the user's original capture slots. The authors developed a Neural Alignment Model trained with a triplet loss objective. This model ensures that "correct" spans in a result sentence are mapped closer to the query arguments in vector space than "distractor" spans.
Key Insights
A major contribution of this work is the General-Purpose Alignment model. Unlike traditional IE systems that require retraining for every new relation, this alignment model is trained once on a diverse set of relations (using the SPIKE-Queries Dataset) and generalizes to unseen queries at test time. This enables "Interactive IE," where users can iteratively build structured tables from massive text corpora without needing a dedicated data science team.
Key Results
The system was evaluated on the CORD-19 dataset, containing millions of scientific sentences. Key findings include:
- Massive Recall Boost: For specific relations like "spreads by," the neural system increased the number of captured facts by 8x (from 5 to 40) with 96% accuracy.
- High Relevancy: Approximately 72.2% of neurally retrieved results were manually verified as relevant across various biomedical relations.
- Alignment Precision: The neural alignment model correctly identified capture slot boundaries 73.8% of the time, demonstrating its ability to handle the inherent "fuzziness" of natural language.
Limitations and Future Work
Despite its success, the system faces challenges. Neural alignment is computationally expensive compared to symbolic search, leading to higher latency. Additionally, the model struggles with numerical sensitivity, often retrieving structurally similar sentences with unrelated numerical values. Future research will focus on a "meta-model" to predict which queries are best suited for neural versus symbolic processing to optimize both speed and quality.
Summary updated