Team GESIS Cologne: An all in all sentence-based approach for FEVER

Wolfgang OttoEMNLP 2018

TL;DR

This paper presents a sentence-based approach for the FEVER (Fact Extraction and VERification) shared task, focusing on retrieving relevant Wikipedia sentences and then classifying their relationship to a given claim. Their novel contribution lies in extending the classification scheme to handle cases where multiple sentences are needed to support or refute a claim, leading to improved performance.

What Problem Does This Paper Address?

The FEVER shared task aims to automatically verify factual claims by finding evidence in Wikipedia. A significant challenge in this task is that a single sentence might not be sufficient to either support or refute a claim. Often, evidence is distributed across multiple sentences, or a claim might be partially supported or refuted by several sentences. Existing approaches often struggle with these multi-sentence evidence scenarios, leading to misclassifications. This paper tackles this by developing a system that can effectively retrieve and classify evidence, even when it's not contained within a single sentence.

Methods

The GESIS Cologne team developed a two-stage pipeline:

  1. Sentence Retriever: This component searches for candidate Wikipedia sentences that are relevant to a given claim. Instead of relying on a traditional document retriever, they directly query all Wikipedia introduction sentences. They use the SOLR search engine, indexing sentences and leveraging noun chunks and named entities extracted from the claim to construct queries. This approach prioritizes sentences that share entities or noun phrases with the claim. To improve retrieval, they preprocess Wikipedia sentences by concatenating the article title to the sentence content, helping to resolve coreferences and maintain context.
  2. Entailment Classifier: This component determines the relationship between a claim and a retrieved sentence. They employ a Decomposable Attention Model, a standard architecture for natural language inference. A key innovation here is the expansion of the classification labels beyond the typical "SUPPORTS," "REFUTES," and "NOT_ENOUGH_INFO." They introduce two new classes: "PART_OF_SUPPORTS" and "PART_OF_REFUTES" to explicitly handle situations where a sentence contributes to a multi-sentence evidence set. For claims labeled "NOT_ENOUGH_INFO," they use the top 3 retrieved sentences to train their classifier, a departure from baseline approaches that used randomly selected sentences.

Finally, an Ensemble Learner (a random forest classifier) aggregates the judgments from the entailment classifier for the top 5 retrieved sentences to produce a single, final judgment for each claim.

Datasets

The paper primarily uses the FEVER dataset for training and evaluation. They mention using the "validation set" for training their ensemble learner and for evaluating their sentence retriever and entailment classifier. They also refer to the "test set" for the FEVER Shared Task.

Key Insights and Ideas

  • Sentence-level processing is sufficient: The authors demonstrate that a pipeline focused solely on individual sentences, without a document retrieval step, can achieve competitive results.
  • Contextualizing sentences with titles is crucial: Prepending Wikipedia article titles to sentences significantly improves the retrieval of relevant information, especially for resolving coreferences.
  • Expanding the label space is vital for multi-sentence evidence: Introducing "PART_OF_SUPPORTS" and "PART_OF_REFUTES" classes directly addresses the limitations of single-sentence classification and is key to handling complex evidence scenarios.
  • Retrieval recall for refuting sentences is lower: The authors observe that their system achieves lower recall for refuting sentences compared to supporting ones, suggesting that refuting sentences might have less direct word overlap with claims.

Key Results

  • The sentence retriever achieved a recall of 53.6% for claims that were fully supported, outperforming the baseline (44.22%) even without a document retrieval component.
  • The entailment classifier achieved an accuracy of 64.7% for sentence-by-sentence prediction across the five classes.
  • The full pipeline achieved a FEVER score of 40.77 on the shared task test set, placing them 8th.

Limitations and Future Work

  • Lower recall for refuting sentences: The authors acknowledge that improving the recall of their sentence retriever for refuting sentences is a key area for future work.
  • Splitting strategies for sentence types: They suggest further research into developing distinct strategies for retrieving and classifying supporting versus refuting sentences.
  • Class imbalance in NOT_ENOUGH_INFO: The paper notes that the "NOT_ENOUGH_INFO" class is over-represented in the validation set, which influences the classifier's precision for that label.

Summary updated

BibTeX
@inproceedings{otto2018team,
  author = {Wolfgang Otto},
  title = {{Team GESIS Cologne: An all in all sentence-based approach for FEVER}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure