Scalable Scientific Interest Profiling Using Large Language Models

Yilun Liang, Gongbo Zhang, Edward Sun, Betina Idnay, Yilu Fang, Fangyi Chen, Casey N. Ta, Yifan Peng, Chunhua Weng

Abstract

Research profiles are essential for surfacing scientific expertise, yet they are frequently outdated due to the manual effort required for maintenance. This study develops and evaluates two large language model (LLM)-based methods to generate scientific interest profiles: one summarizing PubMed abstracts and another utilizing Medical Subject Headings (MeSH) terms. Using a dataset of 595 faculty members from Columbia University Irving Medical Center, we generated profiles using GPT-4o-mini and assessed them through automatic metrics and blinded human review. While lexical overlap with self-written profiles was low (ROUGE-L, BLEU), BERTScore indicated moderate semantic similarity (F1: 0.542 for MeSH-based; 0.555 for abstract-based). In manual evaluations, 77.78% of MeSH-based profiles were rated as good or excellent, and panelists preferred MeSH-based over abstract-based profiles in 67.86% of comparisons. Our findings suggest that LLMs can generate researcher profiles at scale, with MeSH-derived metadata providing a more readable and efficient input than raw abstracts.

Introduction

In the academic ecosystem, a researcher’s scientific interest profile serves as a vital calling card, helping funding agencies identify experts, students find mentors, and scientists discover collaborators. However, these profiles suffer from a "stale profile" problem. Manual updates are time-consuming, and automated web-scraping often results in fragmented or inaccurate information.

This research addresses the challenge of scalable scientific interest profiling by asking whether LLMs can synthesize a researcher’s publication history into a fluent, accurate narrative. By comparing machine-generated summaries against the "ground truth" of self-written biographies, we evaluate the current capabilities of LLMs to bridge the gap between raw bibliometric data and professional narratives.

Methodology

Data Acquisition

The study focused on a primary cohort of 595 faculty members from the Columbia University Irving Medical Center (CUIMC). Publication data, including titles, abstracts, and MeSH terms, were extracted via the NIH Entrez Programming Utilities (E-utilities). To ensure the profiles reflected significant expertise, the pipeline filtered for articles where the researcher was one of the first three authors or the senior author within the last decade.

LLM Strategies

Two primary strategies were implemented using GPT-4o-mini:

  1. Abstract-based (Divide-and-Conquer): To handle scholars with extensive publication records that exceed standard context windows, the researchers used Latent Dirichlet Allocation (LDA) to group publications by topic. These groups were summarized individually before being synthesized into a final narrative profile.
  2. MeSH-based (Keyword-driven): This approach utilized the researcher’s most frequent MeSH terms and keywords as the primary input, prompting the model to generate a narrative based on these standardized descriptors.

Evaluation Framework

Profiles were evaluated using a subset of 167 researchers who possessed existing, human-written online profiles. Evaluation metrics included traditional lexical overlap (ROUGE, BLEU, METEOR), semantic similarity (BERTScore), and TF-IDF Kullback-Leibler (KL) divergence to measure vocabulary distribution differences. Additionally, a blinded panel of human reviewers assessed the profiles for readability and accuracy.

Results

Quantitative Performance

The study highlighted a significant gap between lexical and semantic similarity. Machine-generated profiles showed very low lexical overlap with human-written ones (all metrics below 0.15). However, BERTScore F1 values reached 0.542 for MeSH-based and 0.555 for abstract-based summaries, indicating that the AI captured the underlying meaning of the research even when using different terminology.

Human Preference and Readability

Manual reviews favored the MeSH-based approach. Specifically, 77.78% of evaluators rated MeSH-based profiling as "good" or "excellent." Furthermore, MeSH-based profiles achieved favorable readability ratings in 93.44% of cases and were preferred over abstract-based profiles in 67.86% of direct comparisons. This suggests that "pre-distilled" metadata is a more efficient and less noisy input for LLMs than long-form text.

Discussion

The "Abstraction Gap"

A key finding of the study is the conceptual difference between human and machine summaries. Human-written profiles often feature high-level conceptual abstraction, coining or utilizing synthesized terms like "learning health systems" or "precision medicine." In contrast, LLMs tended to be more literal, focusing on granular methodologies (e.g., "natural language processing") rather than weaving them into a broader scientific vision. This "abstraction gap" suggests that while LLMs are excellent at summarizing what is present in the text, they struggle to emulate the forward-looking, synthesized nature of self-authored biographies.

Topic Persistence

Analysis of topic evolution over a ten-year period revealed that approximately 49% of researchers maintain very stable interests. This "topic persistence" justifies the study's approach of weighting publications equally across a decade rather than focusing exclusively on the most recent work.

Conclusion

Large language models provide a viable path for maintaining up-to-date researcher profiles at scale. While human-written profiles remain the gold standard for conceptual synthesis, MeSH-derived LLM profiles offer high readability and accuracy. Future work may involve the integration of Retrieval-Augmented Generation (RAG) to incorporate external knowledge bases, potentially narrowing the abstraction gap and allowing machine-generated profiles to better reflect the evolving conceptual landscape of scientific research.

Summary updated

Abstract

Research profiles help surface scientists' expertise but are often outdated. We develop and evaluate two large language model-based methods to generate scientific interest profiles: one summarizing PubMed abstracts and one using Medical Subject Headings (MeSH) terms, and compare them with researchers' self-written profiles. We assembled titles, MeSH terms, and abstracts for 595 faculty at Columbia University Irving Medical Center; self-authored profiles were available for 167. Using GPT-4o-mini, we generated profiles and assessed them with automatic metrics and blinded human review. Lexical overlap with self-written profiles was low (ROUGE-L, BLEU, METEOR), while BERTScore indicated moderate semantic similarity (F1: 0.542 for MeSH-based; 0.555 for abstract-based). Paraphrased references yielded 0.851, highlighting metric sensitivity. TF-IDF Kullback-Leibler divergence (8.56 for MeSH-based; 8.58 for abstract-based) suggested distinct keyword choices. In manual review, 77.78 percent of MeSH-based profiles were rated good or excellent, readability was favored in 93.44 percent of cases, and panelists preferred MeSH-based over abstract-based profiles in 67.86 percent of comparisons. Overall, large language models can generate researcher profiles at scale; MeSH-derived profiles tend to be more readable than abstract-derived ones. Machine-generated and self-written profiles differ conceptually, with human summaries introducing more novel ideas.
BibTeX
@article{liang2025scalable,
  author = {Yilun Liang and Gongbo Zhang and Edward Sun and Betina Idnay and Yilu Fang and Fangyi Chen and Casey N. Ta and Yifan Peng and Chunhua Weng},
  title = {{Scalable Scientific Interest Profiling Using Large Language Models}},
  journal = {J. Biomed. Informatics},
  year = {2025},
  url = {https://openreview.net/forum?id=OGAwmrOTng},
}

Loading related content...

Figure