Scalable scientific interest profiling using large language models

Yilun Liang, Gongbo Zhang, Edward Sun, Betina Idnay, Yilu Fang, Fangyi Chen, Casey N. Ta, Yifan Peng, Chunhua Weng

TL;DR

Researchers have developed a scalable pipeline using GPT-4o-mini to automatically generate narrative scientific interest profiles from PubMed data. The study finds that while LLMs struggle with the high-level conceptual abstraction seen in human-written bios, using Medical Subject Headings (MeSH) terms as input produces more readable and accurate profiles than summarizing full publication abstracts.

What Problem Does This Paper Address?

In the academic world, a researcher’s "scientific interest profile" is their calling card. It helps funders find experts, students find mentors, and scientists find collaborators. However, these profiles are notoriously difficult to maintain. Manual updates are time-consuming, and automated web-scraping often results in fragmented, outdated, or inaccurate information.

The authors tackle the challenge of Scalable Scientific Interest Profiling. They ask: Can we use Large Language Models (LLMs) to synthesize a researcher’s entire publication history into a fluent, accurate narrative? And more importantly, how do these machine-generated summaries compare to the "ground truth" of a self-written bio?

Methods

The researchers built a pipeline that extracts publication data (titles, abstracts, and MeSH terms) from PubMed. They focused on "significant contributions"—articles where the researcher was one of the first three authors or the senior author within the last decade.

They tested two primary LLM-based strategies using GPT-4o-mini:

  1. Abstract-based: A "Divide-and-Conquer" approach. Because a senior scholar might have hundreds of papers (exceeding context windows), they used Latent Dirichlet Allocation (LDA) to group publications by topic, summarized those groups, and then synthesized a final profile.
  2. MeSH-based: A keyword-driven approach. Instead of full text, the model was fed the researcher’s most frequent MeSH terms and keywords to generate a narrative.

To evaluate the results, they moved beyond standard metrics like ROUGE or BLEU (which only measure word overlap) and employed BERTScore for semantic similarity and Kullback-Leibler (KL) Divergence to measure how much the "vocabulary distribution" of the AI differed from the human.

Datasets

  • Primary Cohort: 595 faculty members from the Columbia University Irving Medical Center (CUIMC).
  • Evaluation Subset: 167 researchers who had existing, human-written online research profiles to serve as a baseline.
  • Source Data: Metadata, abstracts, and MeSH terms pulled via the NIH Entrez Programming Utilities (E-utilities).

Key Insights and Ideas

  • Keywords > Full Text: Surprisingly, profiles derived from MeSH terms (standardized keywords) were rated higher for readability and overall quality than those derived from full abstracts. This suggests that "pre-distilled" metadata is a more efficient input for LLMs than noisy, long-form text.
  • The "Abstraction Gap": Humans are much better at "coining" concepts. Human-written profiles often included synthesized terms like "learning health systems" or "precision medicine." The LLMs tended to be more literal, often repeating granular methodologies (like "natural language processing") rather than weaving them into a high-level vision.
  • Research Interests are Sticky: By analyzing topic evolution over a decade, the authors found that most researchers (nearly 49%) have very stable interests. This "topic persistence" justifies weighting all publications equally rather than just focusing on the most recent ones.
  • Lexical vs. Semantic Similarity: The study highlights a major pitfall in AI evaluation: machine-generated profiles had very low lexical overlap with human ones (low ROUGE scores) but moderate semantic similarity (BERTScore). This proves that AI can capture the meaning of a researcher's work even if it uses entirely different words to describe it.

Key Results

  • Human Preference: In manual reviews, 77.78% of evaluators rated MeSH-based profiling as "good" or "excellent."
  • The Winner: Panel reviews favored MeSH-based profiles over abstract-based ones in 67.86% of cases.
  • Semantic Alignment: BERTScore F1 values reached 0.542 for MeSH-based and 0.555 for abstract-based summaries. While "moderate," this was significantly higher than traditional lexical metrics (which were all below 0.15).
  • Readability: MeSH-based profiles achieved favorable readability ratings in 93.44% of cases.

Limitations and Future Work

The authors acknowledge several hurdles:

  • Authorship Heuristics: The study relied on a simple rule (first 3 or senior author) to determine "significant" contributions, which might miss important middle-author work in large-scale collaborative fields.
  • Name Disambiguation: While they used institutional affiliations to help, distinguishing between researchers with common names remains a classic "hard problem" in bibliometrics.
  • Domain Specificity: The dataset was limited to a medical center. Future work needs to test if this pipeline holds up in fields like Physics or Humanities where MeSH-like standardized keywords might not exist.
  • RAG Integration: The authors suggest that future iterations could use Retrieval-Augmented Generation (RAG) to pull in external knowledge bases to better emulate the "abstractive" style of human writers.

Summary updated

Abstract

Research profiles help surface scientists' expertise but are often outdated. We develop and evaluate two large language model-based methods to generate scientific interest profiles: one summarizing PubMed abstracts and one using Medical Subject Headings (MeSH) terms, and compare them with researchers' self-written profiles. We assembled titles, MeSH terms, and abstracts for 595 faculty at Columbia University Irving Medical Center; self-authored profiles were available for 167. Using GPT-4o-mini, we generated profiles and assessed them with automatic metrics and blinded human review. Lexical overlap with self-written profiles was low (ROUGE-L, BLEU, METEOR), while BERTScore indicated moderate semantic similarity (F1: 0.542 for MeSH-based; 0.555 for abstract-based). Paraphrased references yielded 0.851, highlighting metric sensitivity. TF-IDF Kullback-Leibler divergence (8.56 for MeSH-based; 8.58 for abstract-based) suggested distinct keyword choices. In manual review, 77.78 percent of MeSH-based profiles were rated good or excellent, readability was favored in 93.44 percent of cases, and panelists preferred MeSH-based over abstract-based profiles in 67.86 percent of comparisons. Overall, large language models can generate researcher profiles at scale; MeSH-derived profiles tend to be more readable than abstract-derived ones. Machine-generated and self-written profiles differ conceptually, with human summaries introducing more novel ideas.
BibTeX
@article{liang2025scalable,
  author = {Yilun Liang and Gongbo Zhang and Edward Sun and Betina Idnay and Yilu Fang and Fangyi Chen and Casey N. Ta and Yifan Peng and Chunhua Weng},
  title = {{Scalable scientific interest profiling using large language models}},
  journal = {J. Biomed. Informatics},
  year = {2025},
  url = {https://openreview.net/forum?id=OGAwmrOTng},
}

Loading related content...

Figure