From Words to Worth: Newborn Article Impact Prediction with LLM

Penghai Zhao, Qinghua Xing, Kairan Dou, Jinyu Tian, Ying Tai, Jian Yang, Ming-Ming Cheng, Xiang Li

TL;DR

Researchers have developed a way to predict the future impact of a scientific paper using only its title and abstract, effectively acting as an automated "double-blind" reviewer. By fine-tuning Large Language Models (LLMs) on a new, time-normalized metric called $TNCSISP$, the system can identify high-impact "newborn" articles without needing to know the authors' reputations or the publication venue.

What Problem Does This Paper Address?

Identifying high-impact research is becoming a "needle in a haystack" problem as the volume of daily submissions (especially on platforms like arXiv) explodes. Traditional Article Impact Prediction (AIP) methods usually rely on "Level I" features: author prestige, venue characteristics (like Journal Impact Factor), or early citation counts.

This creates two major issues. First, these methods are biased toward established researchers and prestigious journals. Second, they can't evaluate "newborn" articles—papers that were just uploaded and have no citation history. The authors propose a shift toward "Level II" prediction, which focuses solely on the internal semantic content of the paper. The goal is to determine an article's "worth" based only on its "words."

Figure 1: A taxonomy of article impact prediction (AIP): since there are virtually no other Lv. II methods, the “new- born AIP” segment represents the proposed approach, which predicts future academic impact in a “double-blind peer- review” manner.

Methods

The authors treat impact prediction as a regression task using decoder-only LLMs (like LLaMA-3 and Mistral).

  1. Metric Innovation ($TNCSISP$): Raw citation counts are poor targets for machine learning because they vary wildly across fields and time. The authors developed the Topic Normalized Citation Success Index - Same Period ($TNCSISP$). This metric represents the probability (0 to 1) that a paper will outperform its peers published in the same field within a specific 6-month window.
  2. Architecture: Instead of using the LLM to generate text, they take the hidden state of the first generated token and pass it through a simple Multi-Layer Perceptron (MLP) with a Sigmoid activation function to output a value between 0 and 1.
  3. Training: They utilized Parameter-Efficient Fine-Tuning (PEFT) via LoRA (Low-Rank Adaptation) and 8-bit quantization to train these models on consumer-grade hardware.
Figure 3: LLM as scholar impact predictor: overall frame- work of the proposed approach. Only the Next Token (first generated token) is used to regress the TNCSISP.

Datasets

The study introduces two new benchmarks:

  • TKPD (Topic Key Phrase Dataset): 251 entries of titles and abstracts manually annotated by AI researchers to identify core research fields. This was used to optimize the prompts that help the system categorize papers.
  • NAID (Normalized Article Influence Dataset): A large-scale dataset of over 12,000 arXiv entries from 2020–2022 across various AI subfields. Each entry includes the title, abstract, and the calculated $TNCSISP$ ground truth.

Key Insights and Ideas

  • Normalization is the Secret Sauce: Predicting raw citations is chaotic for a neural network. By switching the target to $TNCSISP$, the model provides much more stable predictions across different years and disciplines.
  • Semantic Signal: The results suggest that LLMs can actually "understand" the latent features of high-impact work—such as novelty, clarity, and the significance of the claimed contribution—just from the abstract.
  • Scaling Matters: As with most LLM tasks, performance improved steadily as the model parameter count increased (e.g., Qwen-0.5B vs. Qwen-7B).
Figure 4: The impact of various model parameters on per- formance: the larger the number of model parameters, the better the performance.

Key Results

  • State-of-the-Art Performance: The fine-tuned LLaMA-3-8B model achieved a Mean Absolute Error (MAE) of 0.216 and an NDCG@20 (a ranking metric) of 0.901, significantly outperforming traditional LSTM and MLP-based methods that lack external data.
  • Zero-Shot vs. Fine-Tuning: While vanilla ChatGPT (GPT-3.5) struggled to predict impact accurately, the fine-tuned open-source models showed a massive performance leap, proving that specific domain alignment is necessary for this task.
  • Real-World Validation: To test for data leakage, the authors ran the model on 2024 journal articles (published after the training data cutoff). The model’s predicted scores correlated strongly with the journals' JCR quartiles, correctly assigning higher impact scores to papers in Q1 journals.
Figure 6: Predicted TNCSISP values for journals in differ- ent JCR quartiles: higher quartiles show higher predicted values. To avoid potential conflicts of interest, we denote Q1, Q2, Q3, and Q4 to represent articles from journals in JCR quartiles 1, 2, 3, and 4, respectively.

Limitations and Future Work

  • Field Specificity: The current NAID dataset is focused on Artificial Intelligence and Computer Science. The authors acknowledge that the model's performance in vastly different fields (like biology or history) remains to be tested.
  • The "Goodhart's Law" Risk: The authors warn that if such systems become mainstream, researchers might start "optimizing" their abstracts with buzzwords specifically to trick the LLM into giving a higher impact score.
  • Data Scale: The dataset size (12,000 entries) was limited by API constraints. Future work could involve scaling this to millions of papers to further refine the LLM's "intuition."
  • Not a Replacement for Humans: The authors emphasize that this is a screening tool for automated research systems and busy scientists, not a definitive replacement for the peer-review process.

Summary updated

Abstract

As the academic landscape expands, the challenge of efficiently identifying impactful newly published articles grows increasingly vital. This paper introduces a promising approach, leveraging the capabilities of LLMs to predict the future impact of newborn articles solely based on titles and abstracts. Moving beyond traditional methods heavily reliant on external information, the proposed method employs LLM to discern the shared semantic features of highly impactful papers from a large collection of title-abstract pairs. These semantic features are further utilized to predict the proposed indicator, TNCSI_SP, which incorporates favorable normalization properties across value, field, and time. To facilitate parameter-efficient fine-tuning of the LLM, we have also meticulously curated a dataset containing over 12,000 entries, each annotated with titles, abstracts, and their corresponding TNCSI_SP values. The quantitative results, with an MAE of 0.216 and an NDCG@20 of 0.901, demonstrate that the proposed approach achieves state-of-the-art performance in predicting the impact of newborn articles when compared to several promising methods. Finally, we present a real-world application example for predicting the impact of newborn journal articles to demonstrate its noteworthy practical value. Overall, our findings challenge existing paradigms and propose a shift towards a more content-focused prediction of academic impact, offering new insights for article impact prediction.
BibTeX
@inproceedings{zhao2025from,
  author = {Penghai Zhao and Qinghua Xing and Kairan Dou and Jinyu Tian and Ying Tai and Jian Yang and Ming-Ming Cheng and Xiang Li},
  title = {{From Words to Worth: Newborn Article Impact Prediction with LLM}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i1.32106},
}
Figure