Decomposing Compound Figures in Scientific Literature

44 concepts26 papers2 proposals

When training vision-language models on standard datasets, we typically assume a clean, one-to-one mapping: one image paired with one descriptive caption. However, if you open any scientific or medical journal, this assumption immediately falls apart. A vast majority of figures in academic literature are "compound figures"—single image files that multiplex multiple sub-panels, such as a bar chart placed directly alongside a microscopy photo or a series of distinct histological stains.

As Wang et al. and Hsu et al. observe, treating these multi-panel images as single, monolithic units creates a severe bottleneck for automated interpretation systems. To build models that truly understand scientific data, researchers have developed techniques for compound figure decomposition: programmatically breaking down composite graphics into their constituent sub-images and aligning them with their corresponding text segments.

The Problem: The Monolithic Fallacy

Why does treating a compound figure as a single image fail? The core issue is cross-modal semantic noise.

When a model ingests a multi-panel image alongside a single global caption, it encounters a complex "many-to-many" or "one-to-many" mapping problem. According to the ImageCLEFcaption 2017 overview and Monajatipoor et al., a global caption often refers to disparate visual elements simultaneously. For instance, a caption might say: "Panel A shows the control group, while Panel B illustrates the experimental mutation."

If a vision-language model attempts to map that entire text string to the entire composite image, it suffers from two major failures:

  1. Feature Overlap and Confusion: As Rafkind et al. point out, composite figures contain overlapping or conflicting visual features. A model trying to classify the image might get confused by the simultaneous presence of a molecular diagram and a line graph, leading it to incorrectly assign a single, exclusive label based on whichever visual signal happens to be dominant.
  2. Incorrect Associations: The model's contrastive learning mechanism will inadvertently associate the text describing Panel A with the visual features of Panel B. This misalignment introduces massive noise into the training pipeline, degrading the precision of downstream clinical search tools, forensic image analysis, and automated captioning.

The Mechanism: Visual and Textual Disentanglement

To resolve this ambiguity, a decomposition pipeline must independently disentangle both the visual canvas and the textual caption, re-stitching them into clean, isolated pairs.

[ Compound Figure (Panel A | Panel B) ] ----> Visual Segmentation ----> [ Sub-Image A ] [ Sub-Image B ]
                                                                               |             |
[ Global Caption ("A: ... B: ...") ]   ----> Textual Segmentation  ----> [ Sub-Caption A ] [ Sub-Caption B ]

1. Visual Segmentation

Splitting the image canvas requires identifying the boundaries between panels. Early and classical approaches relied heavily on structural heuristics and image processing. For example, Li et al. utilized Connected Component Analysis (CCA) and edge detection to locate white space or grid lines separating panels, while Lee et al. introduced recursive segmentation algorithms to isolate nested visual elements.

This spatial partitioning is also vital outside of scientific literature. In user interface design, Gui employs visual separators and optical character recognition (OCR) to partition monolithic webpage mockups into grid-aligned bounding boxes. Similarly, in digital art, Matsui et al. decompose dense, multi-frame manga layouts into overlapping regions of interest to retrieve specific characters or objects.

Modern biomedical pipelines substitute these rigid rules with deep learning. Frameworks designed by Lin et al. and Moreira et al. deploy object detection networks to automatically predict bounding boxes around individual sub-panels. To bypass the need for expensive manual bounding box annotations, Yao et al. introduced a weakly supervised framework called SimCFS. SimCFS uses a specialized "Side Loss" function during training to prevent over-detection, allowing the network to accurately separate subplots using only single-panel images as weak supervision.

2. Textual Segmentation

Isolating the sub-images is only half the battle; the global caption must also be parsed. Wu et al. demonstrated that rule-based natural language processing can successfully split complex captions by identifying explicit panel markers (such as "(a)", "(b)", or "Left:").

More recently, researchers have integrated large language models (LLMs) into this step. As shown by Zhu et al. and Monajatipoor et al., combining object detection for visual boundaries with LLM-driven text partitioning allows systems to cleanly map localized visual evidence to its exact textual description, resolving the many-to-many mapping problem entirely.

Why It Works: The Intuition of Clean Gradients

The reason compound figure decomposition drastically improves model performance comes down to reducing mutual information noise during training.

When a vision-language model like CLIP is trained on raw, unseparated compound figures, the contrastive learning gradients are pulled in multiple conflicting directions. The text embedding for a specific medical condition is forced to align with an image embedding that contains a mixture of textures, shapes, and chart types. The model essentially tries to learn everything at once, resulting in blurred semantic boundaries.

By decomposing the figure, we transform a high-entropy composite into multiple low-entropy, highly specific training pairs. As Lin et al. demonstrated with PMC-CLIP, pre-training on isolated sub-images and sub-captions ensures that visual features are tightly and accurately coupled with relevant text. This granular alignment prevents the model from hallucinating or misassociating clinical concepts.

A similar intuition applies to image restoration. Niu et al. showed that decomposing a complex image overlay (like a watermark containing both text and logos) into distinct visual components allows specialized sub-networks to handle specific visual properties. Rather than forcing a single network to generalize across heterogeneous data types, decomposition allows models to apply targeted, localized expertise.

Remaining Challenges

While decomposition significantly boosts performance, the process is not without its flaws. Peng notes that automated subfigure separation frequently fails when medical images are densely packed or when high-contrast anatomical features (such as bones or tissue boundaries) are falsely detected as panel borders, resulting in improper cropping.

Furthermore, Liang et al. emphasize that completely separating panels can strip away crucial context, as standard end-to-end neural networks still struggle to model the complex internal spatial dependencies and relationships between panels. As Carr suggests, future research must continue to refine how models segment and classify individual sub-components within highly complex, heterogeneous visual environments without losing the overarching narrative of the graphic.

Key Foundational Work

Exploring Text and Image Features to Classify Images in Bioscience Literature (2006) — contributed 2 concept elements to this direction.
A Technique for Segmentation of Gurmukhi Text (2001) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_compound_figure_decomposition_fd4f1a6d,
  title = {compound figure decomposition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compound-figure-decomposition/rdr_fd4f1a6de5024e08a349abd41e3ef84c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_compound_figure_decomposition_fd4f1a6d_v1,
  title = {compound figure decomposition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compound-figure-decomposition/rdr_fd4f1a6de5024e08a349abd41e3ef84c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake