Compound Figure Separation of Biomedical Images: Mining Large Datasets for Self-supervised Learning

Tianyuan Yao, Chang Qu, Jun Long, Quan Liu, Ruining Deng, Yuanhan Tian, Jiachen Xu, Aadarsh Jha, Zuhayr Asad, Shunxing Bao, Mengyang Zhao, Agnes B. Fogo, Bennett A. Landman, Haichun Yang, Catie Chang, Yuankai HuoarXiv 2022 · DOI · arXiv

Introduction

In the rapidly evolving field of medical artificial intelligence, the availability of high-quality, annotated data remains a primary bottleneck. Self-supervised learning (SSL) has emerged as a powerful strategy to overcome this, allowing models to pre-train on massive, unannotated datasets to learn general features before being fine-tuned for specific clinical tasks. However, a significant portion of biomedical imagery—found in journals, databases, and repositories like NIH Open-i—is stored as compound figures. These are multi-panel collages containing various subplots, such as different histological stains, imaging modalities, or statistical charts.

Standard AI models cannot effectively process these compound images in their raw form; they require individual subplots to be isolated. Manually cropping these figures is labor-intensive and unscalable, while existing automated tools often require their own large sets of manually labeled bounding box data. To address this, we present SimCFS (Simple Compound Figure Separation), a framework designed to automatically and precisely extract subplots, unlocking vast amounts of web-mined data for downstream medical AI applications.

The SimCFS Framework

SimCFS introduces a novel approach to training figure separators by eliminating the need for manual annotations. The framework is built upon three core innovations:

1. SimCFS-AUG: The Simulation-Based Trainer

Instead of relying on human-labeled compound figures, the researchers developed a simulator that programmatically generates training data. By taking individual images mined from the web and stitching them into grids that mimic standard journal layouts, the simulator creates a "ground truth" training set automatically. This allows the model to learn the "grammar" of figure construction—rows, columns, and aspect ratios—without manual intervention.

2. Intra-class Augmentation

One of the most difficult challenges in figure separation occurs when subplots are visually similar and lack clear borders (e.g., two adjacent MRI slices). To handle these "hard cases," the simulator specifically pairs similar images together during the synthesis process. This forces the underlying detection network to identify and learn subtle boundary cues that might otherwise be ignored.

3. Side Loss for Precision

In the context of mining data for SSL, a "tight" crop is significantly more valuable than a "loose" one. Standard object detection models (like the YOLOv5 backbone used here) are often lenient with "over-detection," where bounding boxes are slightly larger than the object. In compound figures, however, an over-detected box captures "noise" or pixels from neighboring subplots, which can degrade the quality of SSL pre-training. The authors introduced Side Loss, a mathematical penalty that specifically punishes the model when the vertices of a predicted box fall outside the actual subplot boundaries, ensuring high-precision extraction.

Experimental Validation

The efficacy of SimCFS was evaluated using a combination of benchmark datasets and real-world clinical applications:

  • ImageCLEF 2016: On this industry-standard benchmark for compound figure separation, SimCFS achieved a state-of-the-art 90.3 [email protected], outperforming both traditional computer vision methods and previous deep learning models.
  • Glomeruli-2000: An in-house dataset of 1,834 real figure plots from the American Journal of Kidney Diseases (AJKD) was used for evaluation. The inclusion of Side Loss and the SimCFS-AUG simulator boosted detection accuracy (mAP) from 69.8 to 80.3 on this challenging renal pathology set.
  • Downstream SSL Utility: To prove the tool's practical value, the researchers used SimCFS to mine 20,000 unannotated glomerular images from the web. They then performed SSL using the SimSiam algorithm. The resulting model achieved an F1 score of 0.900 in a clinical classification task, significantly outperforming a model trained from scratch (0.845) and models pre-trained on the non-medical ImageNet dataset.

Conclusion and Future Work

SimCFS demonstrates that high-quality figure separation does not require exhaustive manual labeling. By leveraging simulation and specialized loss functions, the framework provides a scalable way to transform the vast, untapped archive of compound biomedical figures into a structured resource for AI training.

While highly effective, the authors note that future iterations could further improve by integrating Optical Character Recognition (OCR) to detect sub-figure labels (e.g., "A", "B", "C") and incorporating multi-modal mining to link extracted images with their specific textual descriptions in figure captions. This work represents a significant step toward fully automated, large-scale medical data curation.

Summary updated

Abstract

With the rapid development of self-supervised learning (e.g., contrastive learning), the importance of having large-scale images (even without annotations) for training a more generalizable AI model has been widely recognized in medical image analysis. However, collecting large-scale task-specific unannotated data at scale can be challenging for individual labs. Existing online resources, such as digital books, publications, and search engines, provide a new resource for obtaining large-scale images. However, published images in healthcare (e.g., radiology and pathology) consist of a considerable amount of compound figures with subplots. In order to extract and separate compound figures into usable individual images for downstream learning, we propose a simple compound figure separation (SimCFS) framework without using the traditionally required detection bounding box annotations, with a new loss function and a hard case simulation. Our technical contribution is four-fold: (1) we introduce a simulation-based training framework that minimizes the need for resource extensive bounding box annotations; (2) we propose a new side loss that is optimized for compound figure separation; (3) we propose an intra-class image augmentation method to simulate hard cases; and (4) to the best of our knowledge, this is the first study that evaluates the efficacy of leveraging self-supervised learning with compound image separation. From the results, the proposed SimCFS achieved state-of-the-art performance on the ImageCLEF 2016 Compound Figure Separation Database. The pretrained self-supervised learning model using large-scale mined figures improved the accuracy of downstream image classification tasks with a contrastive learning algorithm. The source code of SimCFS is made publicly available at https://github.com/hrlblab/ImageSeperation.
BibTeX
@misc{yao2022compound,
  author = {Tianyuan Yao and Chang Qu and Jun Long and Quan Liu and Ruining Deng and Yuanhan Tian and Jiachen Xu and Aadarsh Jha and Zuhayr Asad and Shunxing Bao and Mengyang Zhao and Agnes B. Fogo and Bennett A. Landman and Haichun Yang and Catie Chang and Yuankai Huo},
  title = {{Compound Figure Separation of Biomedical Images: Mining Large Datasets for Self-supervised Learning}},
  year = {2022},
  eprint = {2208.14357},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.14357},
}
Figure