Decomposing Compound Figures for Granular Multimodal Understanding

This direction is split into other directions: compound figure decomposition. Its last version stays readable and citable here.

Version 1 of 147 papers68 conceptsfrozen 2026-10-04

Standard vision-language models operate on a simple assumption: one image corresponds to one descriptive caption. While this holds true for a photo of a cat in a park, it fails for the vast majority of scientific and professional communication. In biomedical literature, for example, Lucy Lu Wang and colleagues found in MedICaT that a significant majority of figures are "compound"—complex mosaics containing multiple sub-panels, charts, and microscopy images within a single frame.

When AI treats these compound figures as monolithic blocks, it suffers from "content interference." A model might correctly identify a "cell" and a "bar chart" in the same image but fail to understand which textual description belongs to which visual component. To solve this, a growing body of research focuses on decomposition: the automated process of breaking complex visual layouts into their constituent parts to enable precise, granular reasoning.

The Monolithic Fallacy and Content Interference

The core problem with treating compound figures as single entities is a granularity mismatch. As Ting-Yao Hsu notes in SciCap, scientific captions often provide a holistic summary that describes relationships between panels (e.g., "Panel A shows the control, while Panel B shows the treatment").

If a model like CLIP is trained on the whole image and the whole caption, it learns "noisy" associations. It might associate the word "treatment" with the visual features of the "control" panel simply because they appear in the same training pair. Jing Gu demonstrated this "performance gap" in MultipanelVQA, showing that even powerful models like GPT-4o see their accuracy plummet when individual images are arranged into a grid. The model perceives a "chaotic scene" rather than a structured layout.

Mechanisms of Decomposition

To bridge this gap, researchers employ a "divide and conquer" strategy. This typically involves two parallel pipelines:

  1. Visual Segmentation: Using object detection models to identify "gutters" or separators between panels. Tianyuan Yao and the team behind SimCFS developed a way to train these separators using simulated data, allowing the system to "unmix" compound figures into individual subplots without needing thousands of manually labeled examples.
  2. Caption Partitioning: Using Natural Language Processing (NLP) or Large Language Models (LLMs) to split a long, complex caption into sub-captions. Te-Lin Wu and Nanyun Peng utilized this in the MELINDA dataset, where rule-based segmentation maps specific textual sub-captions to their corresponding visual sub-panels.

By aligning these two streams, researchers transform a single "many-to-many" mapping problem into several "one-to-one" problems, which are much easier for AI to learn.

From Pixels to Hierarchy

Recent work has moved beyond simple splitting toward hierarchical understanding. In Panel2Patch, Kun Yuan 0004 proposes a "zoom-in" approach. Instead of just looking at the whole figure, the model learns to navigate from the full figure to individual panels, and finally to fine-grained "patches" or regions of interest. This hierarchy allows the model to maintain the context of the whole experiment while focusing on the specific evidence in a single corner of the image.

This structural awareness is not limited to science. In web development, Yi Gui and Zhen Li introduced LaTCoder, which uses "Layout-as-Thought." Rather than generating code for a whole webpage at once, the system decomposes the UI mockup into grid-aligned blocks, generates code for each, and then assembles them. This mirrors how humans process complex visuals: we don't see a webpage as a single pixel grid; we see a header, a sidebar, and a content area.

Why Decomposition Works

Decomposition works because it enforces spatial and semantic grounding. When Sheng Zhang developed BiomedCLIP, the goal was to handle the 15 million image-text pairs in the PMC dataset. Without decomposition, the model would be overwhelmed by the "multiplexed" nature of these images—where one figure might contain a flowchart, a table, and a mechanical drawing all at once, as noted by Juan Castorena.

By isolating components, models can:

  • Reduce Noise: Irrelevant visual data from neighboring panels is filtered out.
  • Improve Alignment: Specific claims in a paper can be linked directly to the exact visual evidence that supports them, a task Anukriti Kumar showed is critical for automated scientific evidence identification.
  • Enable Reasoning: Models can perform "closure"—the ability to infer relationships between panels, a concept Mohit Iyyer explored in the context of comic book narratives in the COMICS dataset.

Ultimately, the shift from monolithic to decomposed processing represents a move toward "visual literacy" for AI. It allows models to respect the internal logic and structured grammar of professional imagery, rather than just treating every image as a flat collection of pixels.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_decomposing_compound_figures_for_granula_c10fab22,
  title = {Decomposing Compound Figures for Granular Multimodal Understanding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decomposing-compound-figures-for-granular-multimodal-understanding/rdr_c10fab22ddb1421aadb461047a5accf1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_decomposing_compound_figures_for_granula_c10fab22_v1,
  title = {Decomposing Compound Figures for Granular Multimodal Understanding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decomposing-compound-figures-for-granular-multimodal-understanding/rdr_c10fab22ddb1421aadb461047a5accf1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}