Intelligent Abstractive Summarization of Scholarly Publications with Transfer Learning

Farooq Zaman, Muhammad Zeshan Afzal, Pin Shen Teh, Raheem Sarwar, Faisal Kamiran, Naif Radi Aljohani, Raheel Nawaz, Muhammad Umair Hassan, Fahad SabahJournal of Informatics and Web Engineering 2024 · DOI

This paper, published in the Journal of Informatics and Web Engineering, explores the shift from extractive to abstractive summarization for scientific literature. By comparing traditional Recurrent Neural Networks (RNNs) like LSTMs and GRUs against modern Transformer architectures (BART, PEGASUS), the authors demonstrate that transfer learning and attention mechanisms are essential for generating coherent, human-like titles for complex scholarly articles.

TL;DR

The researchers developed a system to automatically generate titles for scientific papers by "reading" their abstracts. They found that while Gated Recurrent Units (GRUs) are faster and more effective than traditional LSTMs, Transformer-based models—specifically BART—shatter the baseline performance, producing summaries that are significantly more fluent and accurate.

What Problem Does This Paper Address?

As the volume of scientific publications grows exponentially, researchers face a massive "information overflow." Traditional text summarization has relied on extractive methods, which essentially "copy-paste" key sentences from the text. While accurate, these summaries are often clunky, overly long, and lack the flow of human writing.

The authors tackle abstractive summarization, where the model generates entirely new sentences to capture the "gist" of a paper. This is particularly difficult for scholarly text because scientific discourse is compact, technical, and follows a rigid structure (problem, methodology, results, conclusion). The specific task here is "title generation": can a machine read a dense biomedical abstract and produce a concise, meaningful title?

Methods

The study compares three main architectural approaches:

  1. The Baseline: A Long Short-Term Memory (LSTM) encoder-decoder model with an attention mechanism.
  2. The GRU Approach: A Gated Recurrent Unit (GRU) encoder paired with a greedy-search decoder. GRUs are similar to LSTMs but computationally "lighter," designed to solve the vanishing gradient problem more efficiently.
  3. The Transformer Approach (Transfer Learning): The authors fine-tuned three state-of-the-art pre-trained models:
  4. ProphetNet: Uses a novel "n-stream" self-attention to predict multiple future tokens simultaneously.
  5. PEGASUS: Specifically pre-trained for summarization by masking entire sentences.
  6. BART: A denoising autoencoder that learns by reconstructing corrupted text.

The authors applied rigorous preprocessing, including tokenization, replacing all numbers with a hash symbol (#), and filtering data to ensure abstracts were between 150–370 tokens and titles were 6–25 tokens.

Datasets

The researchers utilized the MEDLINE dataset (via a Google repository), which contains metadata for roughly 25 million biomedical papers.

  • Scale: While the full dataset includes 5 million records, the authors used a subset of 100,000 records for training and 1,000 for testing due to the high computational cost of training deep learning models.
  • Format: The models were trained on "title-abstract" pairs, where the abstract serves as the input and the title as the target summary.

Key Insights and Ideas

  • The "Repetition" Trap: One of the most striking insights is the failure of basic LSTMs. The baseline model often fell into "repetitive loops" (e.g., repeating the word "function" or "influence" multiple times), a common failure mode in older RNNs.
  • Speed vs. Performance: The GRU model was 1.5 times faster to train than the LSTM, while the Transformer models were 10 times faster than the LSTM. This highlights that modern architectures aren't just more accurate—they are significantly more efficient.
  • The Power of Pre-training: The "aha moment" lies in the jump in ROUGE scores when moving to BART. By starting with a model that already "understands" English (via transfer learning), the system can focus on the specific nuances of scientific summarization rather than learning grammar from scratch.

Key Results

The models were evaluated using ROUGE-1 (unigram overlap), which measures how many individual words the model got right compared to the human-written title.

  • Baseline LSTM: 0.1033 (Poor performance, high repetition).
  • GRU with Attention: 0.2336 (A significant improvement over the baseline).
  • Transformer (ProphetNet): 0.2881.
  • Transformer (BART): 0.4439 (The top performer).

Qualitatively, the BART and PEGASUS models produced titles that were not only grammatically correct but also captured the specific context of the research (e.g., correctly identifying "age-related changes" in a study about cardiac performance).

Limitations and Future Work

  • Hardware Constraints: The authors explicitly noted that they couldn't use the full 5-million-record dataset because of "colossal memory requirements" and long training times. Future work aims to scale these models to the full dataset.
  • Language Barrier: The current study is limited to English-language papers. The authors plan to expand into multi-lingual summarization in the future.
  • Vocabulary Issues: Handling "Out-of-Vocabulary" (OOV) words—rare technical terms not seen during training—remains a challenge that the team hopes to address by refining their attention mechanisms.

Summary updated

Abstract

Intelligent abstractive text summarization of scholarly publications refers to machine-generated summaries that capture the essential ideas of an article while maintaining semantic coherence and grammatical accuracy. As information continues to grow at an overwhelming rate, text summarization has emerged as a critical area of research. In the past, summarization of scientific publications predominantly relied on extractive methods. These approaches involve selecting key sentences or phrases directly from the original document to create a summary or generate a suitable title. Although extractive methods preserve the original wording, they often lack the ability to produce a coherent, concise, and fluent summary, especially when dealing with complex or lengthy texts. In contrast, abstractive summarization represents a more sophisticated approach. Rather than extracting content from the source, abstractive models generate summaries using new language, often incorporating words and phrases not found in the original text. This allows for more natural, human-like summaries that better capture the key ideas in a fluid and cohesive manner. This study introduces two advanced models for generating titles from the abstracts of scientific articles. The first model employs a Gated Recurrent Unit (GRU) encoder coupled with a greedy-search decoder, while the second utilizes a Transformer model, known for its capacity to handle long-range dependencies in text. The findings demonstrate that both models outperform the baseline Long Short-Term Memory (LSTM) model in terms of efficiency and fluency. Specifically, the GRU model achieved a ROUGE-1 score of 0.2336, and the Transformer model scored 0.2881, significantly higher than the baseline LSTM model, which reported a ROUGE-1 score of 0.1033. These results underscore the potential of abstractive models to enhance the quality and accuracy of summarization in academic and scholarly contexts, offering more intuitive and meaningful summaries.
BibTeX
@article{zaman2024intelligent,
  author = {Farooq Zaman and Muhammad Zeshan Afzal and Pin Shen Teh and Raheem Sarwar and Faisal Kamiran and Naif Radi Aljohani and Raheel Nawaz and Muhammad Umair Hassan and Fahad Sabah},
  title = {{Intelligent Abstractive Summarization of Scholarly Publications with Transfer Learning}},
  journal = {Journal of Informatics and Web Engineering},
  year = {2024},
  doi = {10.33093/jiwe.2024.3.3.16},
}
Figure