TL;DR
This paper introduces a "compounded-label" training strategy for fact-checking systems, merging sentence selection and claim verification into a single training task. This approach significantly reduces training time and computational resources compared to traditional pipeline or multi-task learning methods, while achieving comparable or even superior performance on the FEVER benchmark.
What Problem Does This Paper Address?
Automatic fact-checking is crucial for combating misinformation online. Existing systems often tackle this by breaking it down into sequential sub-tasks: retrieving relevant documents, selecting specific sentences from those documents, and finally verifying a claim against the selected evidence. While effective, this "pipeline" approach is computationally expensive and time-consuming, as each component must be trained separately and sequentially. This makes rapid development and deployment difficult, especially in time-sensitive scenarios. The authors aim to simplify this process and improve efficiency by developing a unified training method.
Methods
The core innovation is the "compounded-label" training strategy. Instead of training separate models for sentence selection and claim verification, the authors propose a single neural semantic matching network (NSMN) that handles both tasks simultaneously. This is achieved by creating a novel "compounded-label space." In this space, the supervision signals for sentence selection and claim verification are merged. For instance, a sentence labeled as "NON-SELECT" for selection is also labeled as "NOTENOUGHINFO" for verification. Conversely, sentences labeled "SUPPORTS" or "REFUTES" for verification are also labeled "SELECT" for selection. This allows the model to learn both tasks jointly from a single, unified objective function, significantly reducing redundancy and computational overhead.
Datasets
The primary dataset used for training and evaluation is FEVER (Fact Extraction and VERification). This dataset is a large-scale benchmark designed for evaluating systems on both evidence retrieval and claim verification. The authors utilize the FEVER 1.0 dataset and its associated challenge.
Key Insights and Ideas
- Unified Supervision is Powerful: Merging supervision signals for related tasks (sentence selection and claim verification) into a single "compounded-label" space allows a single model to learn them more efficiently and effectively than separate training.
- Efficiency Gains are Substantial: The compounded-label approach drastically reduces training time and the amount of data needed for convergence, making it more practical for real-world applications.
- Simplicity Can Lead to Better Performance: By simplifying the training paradigm, the authors achieve performance comparable to, and in some metrics better than, more complex pipeline approaches, while using fewer parameters and less data.
- Natural Regularization: Training on a combined task provides a form of natural regularization, preventing oscillations seen in other multi-task learning setups and leading to a more stable and better global minimum.
Key Results
- FEVER Score: The compounded-label setup achieved a FEVER score of 64.68%, outperforming the multi-task learning setup (62.25%) and direct mixing (57.92%). It also achieved comparable results to the top-performing pipeline setup (65.37%).
- Training Efficiency: The compounded-label approach required significantly less data to reach a functional state. It achieved a 60%+ FEVER score after consuming only 1,280K data points, whereas the pipeline setup needed more data and had a longer initial training phase due to upstream document retrieval.
- Parameter Reduction: The compounded-label setup used one-third fewer model parameters compared to the full pipeline setup.
- Blind Test Results: On the blind test set, the compounded-label setup achieved a FEVER score of 61.65% and a label score (LA) of 66.21%, comparable to the pipeline setup's 62.69% FEVER score and 66.20% LA, but with a simpler design and faster convergence.
Limitations and Future Work
- Document Retrieval Dependency: The performance of the system is still dependent on the quality of the upstream document retrieval stage, which is not the focus of this paper. The authors note that the upper bound for FEVER score is limited by document retrieval quality (92.42%).
- Hyperparameter Tuning: While the compounded-label setup simplifies training, careful hyperparameter tuning (e.g., for the mixing ratios $\lambda_1$ and $\lambda_2$) is still necessary for optimal performance.
- Future Work: The authors suggest that this simplified and efficient training paradigm can facilitate future work on automatic fact-checking and its practical usage, particularly in scenarios requiring rapid deployment or online learning with streaming data. The efficiency gains make it more amenable to real-time human feedback systems.
Summary updated