Hybrid Autoregressive Solver for Scalable Abductive Natural Language Inference

Marco Valentino, Mokanarangan Thayaparan, Deborah Ferreira, André Freitas

Introduction

In the quest for explainable AI, "explanation regeneration" is a high-bar challenge. It requires a model to take a scientific hypothesis (e.g., "Friction causes the temperature of an object to increase") and retrieve a chain of supporting facts from a massive knowledge base to explain why it is true.

Currently, the field is split between two primary approaches:

  1. Cross-encoders: These are highly accurate because they perform deep self-attention between the query and every potential fact. However, they are computationally intensive and do not scale well; searching a million facts this way is prohibitively slow.
  2. Bi-encoders/Sparse Models: These are fast, utilizing pre-computed indexes, but often fail at "multi-hop" reasoning—the ability to link multiple facts together without losing the original context, a phenomenon known as semantic drift.

The authors aim to bridge this gap by creating a system that possesses the reasoning capabilities of a cross-encoder while maintaining the retrieval speed of a bi-encoder.

Methods

The authors introduce SCAR (Scalable Autoregressive Inference). Its architecture relies on three main pillars:

  • Autoregressive Step-by-Step Inference: Instead of retrieving all facts simultaneously, SCAR builds the explanation iteratively. At each step, it concatenates the original hypothesis with previously found facts to search for the next logical component.
  • Hybrid Scoring: SCAR utilizes a joint scoring function that combines:
  • Dense Relevance: Employs Sentence-BERT to identify semantically related facts.
  • Sparse Relevance: Uses BM25 for exact term matching.
  • Explanatory Power: A novel component that analyzes an "Explanations Corpus" to determine if a specific fact is frequently used to explain similar scientific concepts.
  • Efficient Retrieval: By using a bi-encoder architecture, the model leverages FAISS (Facebook AI Similarity Search) to perform Maximum Inner Product Search (MIPS), allowing it to scan millions of facts in milliseconds.
Figure 1: Overview of the SCAR architecture showing the iterative retrieval process and hybrid scoring mechanism.

Datasets

The study primarily utilizes two key resources:

  • WorldTree (V1 & V2): A benchmark dataset of multiple-choice science questions with human-annotated explanation graphs, averaging 6 facts per explanation.
  • GenericsKB: A large-scale knowledge base containing millions of generic scientific and commonsense statements, used to evaluate the model's scalability.

Key Insights and Ideas

  • The "Explanatory Power" Factor: Scientific explanations often reuse "anchor" facts. By explicitly modeling how often a fact explains similar hypotheses, the model can identify high-value "laws" even when semantic overlap is low.
  • Combating Semantic Drift: Multi-hop models often drift away from the topic as the chain lengthens. The authors found that their hybrid approach is significantly more robust than standalone models, maintaining high precision even for explanations requiring 10 or more facts.
Figure 4: Performance comparison (MAP) across different explanation lengths, demonstrating SCAR's robustness against semantic drift compared to bi-encoder and sparse baselines.

Key Results

  • Performance: SCAR achieved a Mean Average Precision (MAP) of 56.22, nearly matching the state-of-the-art cross-encoder (57.07) and significantly outperforming previous sparse models.
  • Speed: SCAR is approximately 50.5 times faster than the leading cross-encoder, processing questions in 0.19 seconds compared to 9.6 seconds.
  • Scalability: In tests with a facts bank expanded to 1 million sentences, SCAR maintained sub-second inference times, whereas a cross-encoder would require hours for a single query.
  • Downstream Success: When used as a solver for multiple-choice questions, SCAR showed improved accuracy and robustness, demonstrating that higher-quality explanations lead to better answers.

Limitations and Future Work

  • The Accuracy Gap: While close, SCAR still slightly trails the best cross-encoders in raw precision, representing the trade-off for removing heavy self-attention mechanisms.
  • End-to-End Training: Currently, the different components are integrated but not trained in a single, end-to-end differentiable pipeline, which remains a key area for future research.
  • Generalization: The "explanatory power" model relies on an existing corpus of explanations. Its performance in domains without such corpora remains to be explored.

Summary updated

BibTeX
@inproceedings{valentino2022hybrid,
  author = {Marco Valentino and Mokanarangan Thayaparan and Deborah Ferreira and André Freitas},
  title = {{Hybrid Autoregressive Solver for Scalable Abductive Natural Language Inference}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21392},
}
Figure