Is the ranking of PubMed similar articles good enough? An evaluation of text similarity methods for three datasets

Mariana Neves, Ines Schadock, Beryl Eusemann, Gilbert Schnfelder, Bettina Bert, Daniel ButzkeACL 2023

Sure, here's a blog-style summary of the paper "Is the ranking of PubMed similar articles good enough? An evaluation of text similarity methods for three datasets":

TL;DR

This paper evaluates the effectiveness of PubMed's "similar articles" feature and compares it against various text similarity methods. The authors found that while PubMed's built-in feature is a useful starting point, several modern text similarity techniques can significantly outperform it, offering more accurate and relevant article rankings for biomedical researchers.

What Problem Does This Paper Address?

Discovering relevant scientific literature is crucial for researchers. PubMed's "similar articles" feature, powered by a probabilistic topic-based model (PMRA), aims to help users by providing a list of articles related to a "seed" article. However, the effectiveness of this feature, especially when compared to a broader range of contemporary text similarity methods, has not been thoroughly evaluated. Researchers often rely on short, keyword-based queries, which can be insufficient for capturing the nuances of their research goals. Seed articles offer a richer context, but their utility depends heavily on the quality of the similarity ranking. This paper seeks to answer: "Is PubMed's similar article ranking good enough, and can we do better?"

Methods

The authors conducted a quantitative evaluation of various text similarity methods. Their approach involved:

  • Text Processing: They used the title and abstract of articles as the primary text source. Additionally, they explored using automatically extracted discourse elements (background, objective, methods, results, conclusions) and manually annotated "facets" (relevant sentences identified by human annotators) to see if these finer-grained text segments could improve similarity scores.
  • Text Representation: A range of methods were employed to represent the text numerically, including:
  • TF-IDF: A classic term frequency-inverse document frequency approach.
  • Word Embeddings: Using domain-specific embeddings like NCBI, BioNLP-EVEX, Cambridge, and ChemPatent.
  • Pre-trained Language Models: Leveraging sentence embeddings from models like BioBERT, PubMedBERT, SPECTER, and SciBERT, among others.
  • Ranking Functions: Cosine similarity was used for most text representations. They also included Okapi BM25, a well-established ranking algorithm. For a subset of their evaluation, they also explored re-ranking based on a combination of PubMed's original ranking and the scores from their tested methods.

Datasets

The study utilized three existing biomedical text similarity datasets:

  • RELISH: A large database with over 180,000 PubMed abstracts validated for similarity to seed articles.
  • TREC-COVID: A dataset focused on COVID-19 publications, used in various information retrieval evaluations.
  • SMAFIRA-c: A smaller dataset comprising four case studies on alternative methods to animal experiments.

They also created "sa-eval" (similar articles evaluation) subsets by considering the overlap between the original dataset's similar articles and those provided by PubMed's similar articles feature at the time of their experiments. This allowed for a direct comparison against PubMed's baseline.

Key Insights and Ideas

  • PubMed's Baseline is Beatable: The most significant finding is that, across all tested datasets and evaluation settings, several text similarity methods consistently outperformed PubMed's built-in "similar articles" ranking. This suggests that while useful, PubMed's feature is not the optimal solution for all similarity ranking needs.
  • Simplicity vs. Complexity: Curiously, both a simple method (TF-IDF) and a more complex, recent method (SPECTER) emerged as top performers for different datasets. This highlights that the "best" method is highly dataset-dependent and not necessarily the most sophisticated.
  • Faceted Information Helps: Using manually annotated "facets" (relevant sentences) for the SMAFIRA-c dataset showed a notable improvement in performance, even with a small number of annotated sentences. This indicates that focusing on specific, relevant parts of an abstract can be more effective than using the entire text.
  • Dataset Characteristics Matter: The performance of methods varied significantly across the three datasets, likely due to their differing sizes, the density of similar articles, and the nature of the similarity annotations. RELISH, with its large size and high rate of similar articles, and SMAFIRA-c, with its smaller scale and specific focus, presented distinct challenges.
  • Re-ranking Offers Mixed Results: While re-ranking based on a combination of PubMed's and other methods' scores showed some improvement, it was not universally beneficial and even degraded performance in some cases.

Key Results

  • Outperforming PubMed: For all three datasets (RELISH, TREC-COVID, and SMAFIRA-c), the study demonstrated that various text similarity methods could achieve better performance than PubMed's baseline ranking.
  • SPECTER and TF-IDF Shine: SPECTER was a top performer for RELISH and TREC-COVID, while TF-IDF excelled for RELISH and SMAFIRA-c.
  • Discourse Elements Show Promise: Using discourse elements (sections like "background," "objective," and "conclusion") generally led to small improvements over using just the title and abstract.
  • Facets Boost SMAFIRA-c: For the SMAFIRA-c dataset, using manually annotated facets significantly improved results, outperforming PubMed's algorithm.

Limitations and Future Work

  • Abstracts Only: The study primarily used article abstracts and did not consider full-text content. While full text could offer more information, its general unavailability in PubMed and the increased complexity of processing it were cited as reasons for this limitation.
  • No Model Training: The authors did not train or fine-tune any of the models, focusing instead on evaluating general-purpose algorithms. This means that models specifically trained on these datasets might achieve even better results.
  • Dataset Annotation Dependency: Two of the datasets (RELISH and SMAFIRA-c) relied on PubMed's similar articles output for their annotations, which could introduce a bias.
  • Future Work: The authors suggest that incorporating full-text analysis, exploring models trained on specific datasets, and further investigating the impact of different text segments (like discourse elements and facets) could lead to even more robust similarity ranking systems. They also hope their work will guide researchers in selecting appropriate methods based on their specific needs and available resources.

Summary updated

BibTeX
@inproceedings{neves2023is,
  author = {Mariana Neves and Ines Schadock and Beryl Eusemann and Gilbert Schnfelder and Bettina Bert and Daniel Butzke},
  title = {{Is the ranking of PubMed similar articles good enough? An evaluation of text similarity methods for three datasets}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure