Abstract
Tables in scientific papers contain crucial information, such as experimental results. Entity Linking (EL) is a promising task that analyses tables and associates them with a knowledge base. EL for table cells requires identifying the referent concept of each cell while understanding the context relevant to each cell in the paper. However, extracting the relevant context from the paper is challenging because the relevant parts are scattered in the main text and captions. This study defines a rule-based method for extracting broad context from the main text, including table captions and sentences that mention the table. Furthermore, we propose synthetic context as a more refined context generated by large language models (LLMs). In a synthetic context, contexts from the entire paper are refined by summarizing, injecting supplemental knowledge, and clarifying the referent concept. We observe this approach improves accuracy for EL by more than 10 points on the S2abEL dataset, and our qualitative analysis suggests potential future works.
Introduction
Scientific papers are dense repositories of data, much of which is structured within tables. To enable automated systems to truly "understand" a paper, they must perform Entity Linking (EL): the process of mapping ambiguous table cells (e.g., "ResNet-50") to specific entries in a structured knowledge base like Papers With Code.
This task is inherently difficult due to three primary factors:
- Scattered Context: Explanations for specific cells are often located far from the table itself, buried in results discussions or brief captions.
- Abbreviations: Technical acronyms like "ET PERP" (Evolved Transformer Perplexity) may appear in tables without their full definitions appearing nearby in the text.
- Semantic Ambiguity: Generic terms such as "Ours," "Baseline," or "All" are common but impossible to link without a deep semantic grasp of the paper's specific methodology.
Methodology
The researchers developed a two-stage pipeline to enrich the context of table cells before linking.
Raw Context Extraction
First, a rule-based system extracts "raw context" from the document. This includes the cell text, its designated type (e.g., "Dataset" or "Method"), the table caption, sentences explicitly referencing the table, and sentences containing the cell's specific text.
Synthetic Context Generation
The core innovation lies in feeding this raw, often noisy context into a Large Language Model (LLM), such as GPT-4 Turbo, to generate a "synthetic context." This refined description serves as a bridge between the paper's internal shorthand and external knowledge.
The Role of Synthetic Context
Synthetic context improves linking through two main mechanisms:
- Context Completion: The LLM expands abbreviations and resolves ambiguous references (e.g., determining that "All" refers to the "entire dataset of TriviaQA").
- Knowledge Completion: The LLM injects external knowledge about well-known methods (e.g., "LSTM") that the author may have assumed the reader already understood.
Experimental Results
The study utilized the S2abEL dataset, a large-scale benchmark for entity linking in machine learning papers, targeting the Papers With Code (PwC) knowledge base. The results demonstrated a significant performance leap:
- Accuracy Boost: The proposed method achieved 70.5% overall accuracy, representing a 12.3-point improvement over the S2abEL baseline of 58.2%.
- In-Knowledge-Base (InKB) Precision: For entities existing within the database, the "hit@1" rate increased from 33.4% to 53.1%.
- Model Versatility: While GPT-4 Turbo yielded the highest performance, open-source models like TULU v2 70B also showed strong results, indicating the approach is not limited to proprietary APIs.
Discussion and Future Work
While synthetic context significantly reduces noise and resolves ambiguity, challenges remain. The authors noted instances of LLM "hallucinations," where the model attributed methods to non-existent papers. Additionally, the system currently identifies "Out-of-KB" entities but cannot yet automatically generate new entries for novel concepts. Future research will focus on expanding this methodology to other complex domains, such as biology and chemistry, and mitigating model hallucinations.
Summary updated