TL;DR
This paper introduces a data-centric approach to improve entity linking (EL) in knowledge-based question answering (KBQA), focusing on short text. By developing a "Domain information Mining and Explicit expressing" (DME) model to enrich data with domain-specific information and a novel negative sampling strategy, the authors significantly boost the performance of existing EL models without altering their architecture.
What Problem Does This Paper Address?
Entity linking (EL) is a crucial step in knowledge-based question answering (KBQA), aiming to connect mentions in natural language questions to entities in a knowledge base (KB). While much research has focused on EL in long texts, open-domain KBQA often deals with short, ambiguous questions. Existing models often rely on complex neural network architectures and struggle with adaptability across different datasets, especially for less mature KBs like those in Chinese. The authors identify a need for a more data-centric approach that enhances the input data itself, rather than solely focusing on model complexity, to improve EL performance and robustness, particularly for short text.
Methods
The paper proposes two main contributions:
- Domain Information Mining and Explicit Expressing (DME) Model: This data-centric approach aims to enrich the input data by extracting and explicitly representing domain information from short text. DME identifies feature words in a question or entity description and uses domain dictionaries to calculate a "domain contribution score." This score, combined with prior probabilities of domains, helps determine the most relevant domains for the query. This enriched domain information is then used as an additional input for the EL model.
- Novel Negative Sampling Strategy: To improve the robustness of EL models, the authors developed a negative sampling approach that considers both the surface form (edit distance) and semantic similarity (cosine distance using BERT embeddings) of mention-entity pairs. This strategy aims to generate "hard" negative samples that better train the model by extending the decision boundary, leading to improved classification accuracy.
Datasets
- KgCLUE: A large Chinese open-source KBQA benchmark used for training and evaluation. It includes a KB, QA dataset, and baseline models.
- NLPCC2016: A dataset with more noise than KgCLUE, used to test the adaptability of the proposed method.
Key Insights and Ideas
- Data-centricity over Model-centricity: The core idea is that improving the quality of the input data can significantly enhance EL performance, even for existing, complex models, without requiring architectural changes.
- Domain Information is Key for Short Text: For short, ambiguous queries, explicit domain information is vital for accurate entity linking. DME effectively captures this by leveraging domain dictionaries.
- "Hard" Negative Samples Improve Robustness: Simply using random negative samples is insufficient. A strategy that balances surface form and semantic similarity for negative samples leads to better-trained models and improved decision boundaries.
- Transferability: The proposed DME approach demonstrates good transferability to different datasets (KgCLUE and NLPCC2016), highlighting its general applicability.
Key Results
- Performance Improvement: Fine-tuned BERT models using DME-processed data showed significant improvements, with an increase of about 7% in accuracy and F1 scores on the KgCLUE dataset compared to models trained on original data.
- Adaptability: DME also improved BERT's performance on the NLPCC2016 dataset, demonstrating its suitability for different datasets.
- Negative Sampling Effectiveness: The proposed negative sampling strategy outperformed random sampling, achieving higher accuracy and F1 scores. It also reduced training time overhead.
- Optimal Negative Samples: Experiments suggested that using one positive sample with two negative samples optimally trains the model.
- Ablation Studies: Experiments confirmed that DME, the proposed negative sampling strategy, and their combination all contribute to improved entity linking performance.
Limitations and Future Work
- Limitations: The authors acknowledge that the effectiveness of DME can be influenced by the quality of the domain dictionaries, and errors in domain recognition can occur. The limited datasets and models used for testing also present a limitation in fully verifying the approach's transferable ability.
- Future Work:
- Improving domain dictionary quality through expert inspection to increase DME's accuracy.
- Proving the effectiveness of the proposed method on more recently proposed advanced models.
- Extending the DME approach to other tasks, such as text classification and search engines.
Summary updated