The Search for Conflicts of Interest: Open Information Extraction in Scientific Publications

Garima Gaur, Oana Balalau, Ioana Manolescu, Prajna Devi UpadhyayEMNLP 2025

TL;DR

This paper introduces a novel pipeline for automatically extracting "Conflicts of Interest" (COI) relationships from scientific publications using Large Language Models (LLMs). By combining LLM-based extraction with rule-based refinement and clustering, they build a comprehensive knowledge base of COI disclosures, revealing significant differences in industry involvement between two toxicology journals.

What Problem Does This Paper Address?

In many critical decision-making processes, expert input is crucial. To ensure impartiality and public trust, it's vital that these experts do not have "Conflicts of Interest" (COIs) – situations where their personal or financial interests could bias their judgment. While transparency databases exist to catalog these relationships, manually extracting and organizing COI information from scientific literature is a massive undertaking. Existing methods often struggle with the diverse ways COIs are disclosed in scientific papers, which can appear in metadata, acknowledgments, or funding sections, and can be expressed in various linguistic forms. This paper aims to bridge this gap by developing an automated system to extract these complex relationships from scientific texts.

Methods

The core of the paper's methodology is an "Open Information Extraction" (OpenIE) pipeline that leverages the power of LLMs. Here's a breakdown:

  • LLM-based Triple Extraction: They use LLMs (specifically GPT-3.5 and Gemma 2 models) with few-shot prompting to extract atomic triples (subject, predicate, object) from disclosure texts. This approach is chosen for its ability to handle the nuanced language of scientific disclosures.
  • Triple Refinement: The raw triples extracted by LLMs are often noisy. The system employs Named Entity Recognition (NER) to filter triples, ensuring that subjects and objects are valid entities (Person, Organization, or Grant). They also implement rules to handle implicit subjects and objects, making the extracted information more robust.
  • Predicate Normalization: To create a structured knowledge base, predicate phrases are normalized. This involves identifying a "base predicate" that captures the core meaning and then clustering these base predicates to form a manageable vocabulary of normalized relations. This step uses LLMs for base predicate extraction and hierarchical agglomerative clustering for normalization.
  • Entity Normalization: Author names are anonymized with numeric identifiers, and organizations are normalized by handling abbreviations and clustering similar entities to ensure consistency.
  • Knowledge Base Construction: The pipeline culminates in building a knowledge base (KB) containing over 30,000 triples from scientific articles.

Datasets

  • RTP and TAP Journals: The primary dataset consists of publications from two journals in the toxicology and pharmacology field: "Regulatory Toxicology and Pharmacology" (RTP) and "Toxicology and Applied Pharmacology" (TAP). Data was collected over a 20-year period (1993-2022).
  • Existing Gold Standard: For evaluating triple extraction, they utilized a gold set proposed by Upadhyay et al. (2023), which comprises 1113 pairs of (sentence, triple) annotated from PubMed disclosure statements.

Key Insights and Ideas

  • LLMs as Powerful OpenIE Tools: The research demonstrates that LLMs, even with few-shot prompting, can achieve competitive performance in OpenIE tasks, particularly when combined with careful prompt engineering and post-processing.
  • The Importance of Implicit Disclosures: A significant finding is that while explicit COI metadata is rare (around 1% of papers), a vast majority (around 99%) contain relevant phrases within the PDF text, highlighting the necessity of extracting information beyond structured metadata.
  • Nuance in COI Relations: The paper emphasizes the need for a rich, nuanced vocabulary to capture the complexities of COI relationships, going beyond simple categories. Their developed vocabulary of over 300 predicates aims to address this.
  • Journalistic Practices Inform Research: The work is inspired by journalistic investigations into COIs, suggesting a strong link between academic research and real-world impact.

Key Results

  • High-Quality Triple Extraction: The proposed pipeline, particularly when using GPT-3.5, achieved strong performance on triple extraction tasks, outperforming baseline methods. For instance, on the Carb metric, GPT-3.5 achieved a precision of 0.761 and a recall of 0.852.
  • Comprehensive COI KB: They successfully built a knowledge base containing over 30,000 COI-related triples.
  • Striking Journal Differences: A key qualitative finding is the vastly different levels of industry involvement between RTP and TAP journals. RTP showed significantly higher company involvement in both author affiliations and COI statements compared to TAP. For example, company employers were nine times more present in RTP author affiliations.

Limitations and Future Work

  • PDF Text Extraction Challenges: Errors in extracting text from PDFs, especially from the end of articles where acknowledgments are often located, can lead to noise and incorrect extractions.
  • LLM Brittleness: LLMs can sometimes confuse syntactically similar but semantically opposite phrases, requiring robust post-processing.
  • Journal Classification: The current system might misclassify journals as organizations if they appear in COI statements. Future work could involve extending the LLM classification to specifically identify journals.
  • Broader COI Taxonomy: While their vocabulary is rich, the authors note that existing COI taxonomies might lack the richness to capture all nuances, suggesting a need for further refinement and expansion.
  • Automated Weeding: Automating the removal of irrelevant extractions (like "FDA, has headquarters at, Maryland") is an area for future improvement.

Summary updated

BibTeX
@inproceedings{gaur2025search,
  author = {Garima Gaur and Oana Balalau and Ioana Manolescu and Prajna Devi Upadhyay},
  title = {{The Search for Conflicts of Interest: Open Information Extraction in Scientific Publications}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure