Semantic Matching Against a Corpus: New Applications and Methods

Lucy H. Lin, Scott B. Miles, Noah A. SmitharXiv 2018 · OpenReview

TL;DR

This paper introduces "semantic matching against a corpus," a new task for finding sentences in a large text collection that express a specific natural language idea. They demonstrate its utility in disaster recovery scenarios and develop new methods, including an entailment-based model, that outperform simple word-vector averaging for this challenging task.

What Problem Does This Paper Address?

Imagine you're a historian wanting to track the discourse around a specific scientific claim, a political scientist analyzing nuanced public opinion on immigration, or a disaster recovery manager trying to understand community challenges. All these scenarios require finding sentences in a vast amount of text that express a particular idea, not just keywords. This is the core problem: how to semantically match a natural language "proposition query" (an idea) against a corpus of text. Existing methods often focus on exact matches or broader semantic similarity, but this paper targets the more nuanced task of finding sentences that express the same underlying meaning as a given idea, even if the wording is very different.

Methods

The authors explore a spectrum of methods, starting with simpler approaches and progressing to more sophisticated ones.

  • Word Vector Averaging: This baseline method represents each sentence as the average of its word vectors (using pre-trained embeddings like Word2Vec or paraphrastic vectors). Similarity is then calculated using cosine similarity. They also test a TF-IDF baseline as a sanity check.
  • Tree Edit Models: To capture more nuanced semantic relationships, they adapt tree edit models, which work by comparing the dependency parse trees of sentences. They implement two variants:
  • Logistic Regression (LR) Tree Edit Model: This model uses features extracted from the edit operations needed to transform one sentence's parse tree into another's.
  • Neural Tree Edit Model (LSTM): This approach uses a Long Short-Term Memory (LSTM) network to process the sequence of tree edit operations, allowing it to capture sequential and lexical information more effectively.
  • Fast Filter: To handle large corpora efficiently, they propose using the word-vector averaging models as a "fast filter" to quickly identify a smaller set of candidate sentences, which are then re-ranked by the more computationally expensive entailment models.

Datasets

The paper utilizes and adapts existing datasets for evaluation:

  • CNN/Daily Mail Reading Comprehension Dataset: Used to create a task where the goal is to identify the sentence in an article that contains a specific entity from a query.
  • Media Frames Corpus: Adapted to create a task where proposition queries are derived from expert-designed examples of "framing dimensions" in news articles about immigration.
  • Stanford Natural Language Inference (SNLI) Corpus: Used for training the entailment-based models. The premise-hypothesis pairs are treated as candidate-query pairs, with entailment labels used for positive matches.
  • New Zealand Disaster Recovery Corpus: A custom corpus of 982 articles related to the Canterbury/Christchurch earthquakes, collected and annotated for a more realistic application.

Key Insights and Ideas

  • Disaster Recovery is a Realistic Use Case: The paper highlights that domain experts, like emergency managers, have a genuine need for tools that can surface relevant information from large text collections based on specific ideas.
  • Entailment is Key, But Not Sufficient: While entailment models show improvement over simple word-vector averaging, the authors find that the disaster recovery task requires more than just detecting sentence-level entailment. The nuances of domain-specific language and context are crucial.
  • Paraphrastic Vectors Offer an Edge: The paraphrastic word vectors, specifically trained for semantic similarity, consistently yielded better results than general-purpose Word2Vec embeddings in the word-vector averaging baselines.
  • User Studies are Crucial: The work emphasizes the importance of user studies with domain experts to evaluate the quality of semantic matching results, as objective metrics alone may not fully capture utility.
  • Semantic Measurement is a Promising Application: Beyond just finding matches, the paper introduces "semantic measurement" – tracking the frequency of an idea over time – as a valuable application, demonstrated with an example from the disaster recovery domain.

Key Results

  • Word Vector Baselines are Encouraging: For preliminary tasks using existing corpora, word-vector averaging models achieved reasonable recall and precision, outperforming TF-IDF.
  • Entailment Models Improve Disaster Recovery Matches: In the disaster recovery user study, the tree edit models (both LR and LSTM variants) provided higher-quality matches than the word-vector averaging baselines. The LSTM version showed a slight edge over the LR model.
  • User Study Scores: Average user scores for the top-ranked sentences from the word-vector averaging models (reranked by tree edit models) were around 2.7-3.1 on a 1-5 scale, with paraphrastic vectors generally leading to higher scores.
  • Follow-up Study Replicates Findings: A follow-up study with user-generated queries confirmed the relative performance benefits of the proposed models.
  • Post-hoc Evaluation Shows Challenge: While state-of-the-art entailment models (DAM, ESIM) performed well on SNLI, their performance on the disaster recovery data was significantly lower, indicating the task's difficulty.

Limitations and Future Work

  • Sentence-Level Segmentation: The current approach segments text at the sentence level. Future work could explore matching across larger passages or even within phrases, potentially incorporating coreference resolution and entity linking.
  • Domain-Specific Entities: The paper notes that domain-specific entities (like government agencies) can be challenging if they lack good distributed representations or are not well-represented in training data.
  • Granularity of Matches: The desired granularity of matches can vary. For some applications (like framing), very general propositions might be too broad for current models to detect reliably.
  • Computational Cost: Sophisticated entailment models are computationally expensive. The fast filter approach helps, but further optimization is always beneficial.
  • Beyond Entailment: The authors suggest that future work might need to go beyond simple sentential entailment to capture the complex semantic relationships required for some applications.

Summary updated

Abstract

We consider the case of a domain expert who wishes to explore the extent to which a particular idea is expressed in a text collection. We propose the task of semantically matching the idea, expressed as a natural language proposition, against a corpus. We create two preliminary tasks derived from existing datasets, and then introduce a more realistic one on disaster recovery designed for emergency managers, whom we engaged in a user study. On the latter, we find that a new model built from natural language entailment data produces higher-quality matches than simple word-vector averaging, both on expert-crafted queries and on ones produced by the subjects themselves. This work provides a proof-of-concept for such applications of semantic matching and illustrates key challenges.
BibTeX
@misc{lin2018semantic,
  author = {Lucy H. Lin and Scott B. Miles and Noah A. Smith},
  title = {{Semantic Matching Against a Corpus: New Applications and Methods}},
  year = {2018},
  eprint = {1808.09502},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.09502},
}
Figure