Bag of concepts can enhance other text representations for medical systematic review screening

Radosław Pytlak, Paweł Cichosz, Bartłomiej Fajdek, Bogdan JastrzębskiarXiv 2026 · arXiv

In the high-stakes world of Evidence-Based Medicine (EBM), Systematic Literature Reviews (SLRs) are the gold standard. However, they are a researcher's nightmare: screening 7,000 abstracts to find a handful of relevant studies can take a week of grueling expert labor. While Large Language Models (LLMs) have revolutionized text processing, a new paper suggests that old-school domain knowledge—specifically "Bag of Concepts"—might be the secret sauce needed to make these systems truly efficient.

TL;DR

By augmenting modern neural embeddings (like Qwen3 and PubMedBERT) with "Bag of Concepts" (BoC) features derived from medical ontologies, researchers can significantly increase workload savings in systematic review screening. The study finds that these structured domain features provide complementary information that even the most powerful LLMs don't fully capture on their own.

What Problem Does This Paper Address?

The core challenge in semi-automated SLR screening is text representation. How do we turn a medical abstract into a vector that a machine can classify as "relevant" or "irrelevant"?

Traditional methods used Bag of Words (BoW), which looks at raw word frequencies. Modern methods use deep learning embeddings (like BERT or LLMs), which capture semantic context. However, medical language is highly specialized. A "heart attack" and "myocardial infarction" are the same concept but different words. While LLMs are good at picking up these relationships, they can still struggle with the precise, hierarchical nature of medical taxonomies. The authors investigate whether explicitly mapping text to standardized medical concepts (BoC) can enhance these modern representations.

Methods

The researchers employed an Active Learning (AL) framework. In this setup, a machine learning model (Random Forest) starts with a small set of labeled abstracts and iteratively "asks" a human expert to label the most uncertain cases, rapidly improving its accuracy.

The key innovation is the augmentation strategy. They compared and combined several representations:

  • Lexical: Bag of Words (BoW).
  • Concept-based: Bag of Concepts (BoC) using ontologies like SNOMED CT and MeSH. They used annotators like NOBLE and MedCAT to map text to these concepts.
  • Neural Embeddings: PubMedBERT (domain-specific), Jina, Gemma, and Qwen3 (general-purpose LLM embeddings).

They tested "augmented" versions where a base representation (e.g., Qwen3) was concatenated with BoC features to see if the combination outperformed the parts.

Datasets

The study utilized 15 medical SLR case studies from the Drug Effectiveness Review Project (DERP). These datasets cover a wide range of topics, from ADHD to Beta Blockers, and are characterized by significant class imbalance (very few relevant papers compared to the total).

Table 1: Test cases.

Key Insights and Ideas

  • Complementary Strengths: Neural embeddings are excellent at capturing the "vibe" and context of a sentence, but BoC features act as a "hard" anchor to verified domain knowledge. The results show that adding BoC improved performance in almost every case.
  • The "Short" Variant Wins: Interestingly, the authors found that complex hierarchical expansion (adding parent concepts from an ontology) wasn't always better. A "short" BoC representation—which simply prunes very rare or very common concepts—was often the most effective and computationally efficient.
  • Ontology Sensitivity: The choice of ontology (e.g., MeSH vs. SNOMED) matters. Performance varied across different medical topics, suggesting that a "one-size-fits-all" ontology doesn't exist for all of medicine.
  • Feature Density: As seen in the feature counts, BoC representations often result in more features than BoW, providing a richer (though sparser) signal for the classifier.
Table 2: The number of features in the BoW, BoC (s) representations (S+M–SNOMED CT + MeSH, C+N–CheBI + NCI Thhesaurus, Med–MedCAT).

Key Results

The primary metric was Workload Savings at 95% Yield (WSS@95%), which measures how much human effort is saved while still finding 95% of all relevant articles.

  • Top Performer: The Qwen3 + BoC combination achieved the best average performance across the board.
  • Consistent Gains: Augmenting base models with BoC features led to higher mean workload savings in 10 or 11 out of the 15 case studies, depending on the base model.
  • Quantitative Boost: For Qwen3, adding BoC features increased the mean WSS@95% by 0.0252, a significant margin in the context of high-volume screening.
Table 3: Paired comparison of mean WSS@95% for base and BoC-augmented text representations.

Limitations and Future Work

While the results are promising, the authors note a few areas for further exploration:

  • Hyperparameter Tuning: The study used mostly default settings for the Random Forest classifier to simulate "real-world" conditions where labels are too scarce for extensive tuning. Optimized models might show even greater gains.
  • LLMs as Classifiers: This study used LLMs to generate embeddings for a separate classifier. Future work could compare this to using LLMs directly as zero-shot or few-shot classifiers.
  • Domain Generalization: The study focused on medical abstracts. Whether BoC-style augmentation works as well in other technical fields (like law or engineering) remains an open question.

Summary updated

Abstract

The paper examines the choice of text representation for semi-automated systematic literature review (SLR) systems based on active learning. Motivated by the insufficient attention that the bag of concepts representation has received so far, the paper discusses the implementation and experimental evaluation of several variations thereof. The experimental study was performed on medical SLR case studies, with results reported for random forest classification, cluster initialization and uncertainty sampling. The utility of bag of concepts features obtained using different ontologies is evaluated by applying them within the active learning framework to sequentially create classification models based on training data iteratively expanded with expert annotations. Systematic experiments compare BoW, BoC, PubMedBERT, Jina, Gemma and Qwen3 representations, together with combinations of BoC and each of the other representations. The evaluation uses workload savings and performance profiles adapted from optimization benchmarking. The results suggest that concept-based features can provide complementary information, with the BoC-augmented representations achieving higher mean workload savings on the DERP cases than their corresponding base representations. The Qwen3+BoC representation gives the best average performance in this comparison. The effects of augmentation vary across case studies and base representations. The paper also concentrates on the impact of ontology choice on document classification outcomes. In particular, the paper compares the predictive value of bag of concepts representations based on chosen ontologies with that of representations obtained using MedCAT with a UMLS-based dictionary.
BibTeX
@misc{pytlak2026bag,
  author = {Radosław Pytlak and Paweł Cichosz and Bartłomiej Fajdek and Bogdan Jastrzębski},
  title = {{Bag of concepts can enhance other text representations for medical systematic review screening}},
  year = {2026},
  eprint = {2611.15798},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15798},
}
Figure