Seam Evidence for Compound Figure Splitting

Scientific papers often pack several images into one figure: a microscopy photo next to a gel blot next to a chart, all under one caption. Before these figures can train vision-language models or feed image search, software has to cut them into their separate panels. The hard cases have no white gutter between panels. Two adjacent kidney-tissue photos can share the same stain and the same texture. When the cut is a few pixels too loose, pixels from one panel leak into the crop of its neighbor and corrupt downstream learning. Current detectors find these seams only implicitly from raw pixels. The most accurate recent system runs a 7-billion-parameter vision-language model per figure, which is costly when the target is millions of figures.

This proposal starts from one observation. Inside a single panel, lighting and staining change smoothly, because the panel is one photograph of one scene. At a seam, the image stops being explainable as one smoothly lit scene. Either the texture pattern breaks, or the overall brightness gain jumps, or both. We propose to measure both signals cheaply. Texture breaks are measured with a lighting-invariant local pattern code, chosen so that vignetting and stain gradients inside a panel do not register as edges. Gain jumps are measured as steps in log-intensity. The combined map is then used in two places: as extra input channels to a small detector, and in a test-time step that snaps each predicted box edge to the strongest seam nearby. Because panels are axis-aligned rectangles, snapping is an exact one-dimensional search per edge rather than an iterative optimization loop, so it adds almost no time.

Core Claim

Our claim is that panel boundaries in seamless compound figures are best found by separating two kinds of image change. The first is smooth photometric change (lighting, staining, vignetting), which should be ignored. The second is texture or gain discontinuity, which marks a seam. We claim that an explicit, lighting-invariant seam map, used for one-dimensional box-edge snapping, makes boxes tighter than a detector trained with a boundary penalty alone. We expect this to show most clearly at strict overlap thresholds and under lighting shifts the detector never saw. The method should run at lightweight-detector speed, not vision-language-model speed.

For a candidate vertical cut at column $c$ inside a predicted box, define the seam evidence

$$E(c) = d\big(h^{-}_c, h^{+}_c\big) + \lambda \,\big|\log \mu^{+}_c - \log \mu^{-}_c\big|$$

where $h^{-}_c$ and $h^{+}_c$ are histograms of local pattern codes in narrow strips of width $w$ immediately left and right of $c$ (restricted to the box's row span), $d$ is a chi-squared distance, $\mu^{\pm}_c$ are the mean intensities of those strips, and $\lambda$ weights the gain-step term. Horizontal cuts are handled the same way. Snapping replaces a predicted edge $c_0$ with $c^* = \arg\max_{|c-c_0|\le r} E(c)$, where $r$ is a search radius.

Each term covers a different failure. The pattern codes are invariant to multiplicative lighting, so a smooth vignette contributes almost nothing to the first term. In log space a multiplicative illumination field becomes additive, so a smooth field adds only about $w$ times its gradient to the second term. A true seam between separately photographed panels produces a jump in either term.

Relation to Prior Work

Three groups of prior work bear on the claim, and each plays a different role.

Work that motivates it. The SimCFS line (Compound Figure Separation of Biomedical Images with Side Loss; its expanded version) synthesizes training figures by stitching single images. It uses "intra-class augmentation", pairing similar images such as adjacent tissue slices, to force a YOLOv5 detector to learn faint seams. It adds Side Loss, a penalty on box edges that overshoot the true panel.

  • Its ablation is our motivating evidence: on Glomeruli-2000, removing these ingredients drops mAP from 80.3 to 66.4 (69.8 in the 2022 version). Boundary precision is therefore the bottleneck.
  • SimCFS learns seams only implicitly from raw pixels. We add an explicit invariant cue and a test-time correction.

Work that constrains the design. The classical NLM pipeline (NLM at imageCLEF2015) detects gradient lines and uniform "crossing bands". It reached 84.64% accuracy on ImageCLEF 2015 but needs figures pre-sorted into stitched and gutter types. It is confused by chart axes and uneven backgrounds, which tells us that raw-gradient line evidence is not enough.

Work that supplies the method. The pattern code is the scale-invariant local ternary pattern of Liao et al.. For each neighbor it records "brighter by more than a factor $1+\tau$", "darker by more than $1-\tau$", or "similar".

  • This gives exact invariance to multiplicative scaling.
  • The tolerance band suppresses the noise-driven bit flips that make plain local binary patterns brittle in flat regions.
  • It runs at 63.9 FPS on a CPU for video background subtraction.
  • It has never been applied to static spatial seam finding, which is our transfer.

For stained histology we optionally compute codes on the hematoxylin optical-density channel. This follows the color deconvolution assessment, which studied stain variability only on pre-cropped patches.

Work that sets the speed regime. The latency literature shows that slow per-instance optimization can usually be replaced by a single feed-forward pass. Texture Networks cut texture synthesis from about 10 s to 20 ms, AdaIN runs arbitrary style transfer in 0.02 s, and SwiftSketch is roughly 600x faster than optimization-based sketching. We take that lesson as a constraint rather than a method. Boundary refinement for panels should not be an iterative active-contour or energy-minimization loop. The rectangle structure lets us solve it exactly in one pass.

The accuracy reference. FigEx joins a 7B vision-language model to a DeiT detector and reports AP 0.753 on BioSci-Fig, 0.108 above YOLO11. It is the accuracy ceiling we compare against, at a very different cost.

Research Program

Stage 1: the seam map. We build the map from the pattern codes at two or three tolerance values $\tau$ and strip widths $w$. Both the texture-distance and log-gain terms are computed with integral histograms, so evaluating $E(c)$ over all columns of a row span costs time linear in image size.

Stage 2: two ways to use the map. - As input. The two seam terms are appended as extra input channels to the SimCFS YOLOv5 detector, trained on SimCFS-AUG synthetic figures. - As a test-time step. Each predicted edge is snapped to the arg-max of $E$ within radius $r$. If no peak clears a margin over the local median, the edge is left unchanged, so snapping cannot make a confident box worse in flat regions.

Stage 3: a controlled stress test. At test time we add per-panel gain jitter, page-level illumination gradients, vignetting, and H&E stain shifts. Their ranges deliberately exceed anything used in training augmentation. This separates invariance that holds by construction from invariance that was learned.

Baselines

Naive reference. A YOLOv5 detector trained on SimCFS-AUG data without Side Loss or intra-class augmentation. Yao et al. report this at 66.4 mAP on Glomeruli-2000 (69.8 in the 2022 paper). Any proposed arm that fails to clear it clearly has failed outright.

Strongest directly comparable method. SimCFS with Side Loss and intra-class augmentation, reproduced from the released code (hrlblab/ImageSeperation). Reported results are 80.3 mAP on Glomeruli-2000 and 88.9 (2021) or 90.3 (2022) [email protected] on ImageCLEF 2016.

  • Why it is the right competitor. It attacks exactly our question, tight boxes on seamless similar-texture panels, at the same model size.
  • Fair-comparison rule. We retrain it with the same simulator data, the same photometric augmentation given to our arms, the same epochs and resolution, and the same hyperparameter search budget.
  • What counts as beating it. A statistically reliable gain (bootstrap over test figures) at [email protected] and [email protected], where boundary bleed matters and [email protected] hides it. In addition, no loss at [email protected], and a smaller drop under the stress test.

High-capacity reference. FigEx-7B on BioSci-Fig, with reported AP 0.753 and YOLO11 0.108 below it. If released weights are unavailable we compare against the reported number only. The file reports no throughput figure; there is no reported result found on Lacuna for FigEx latency, so we will measure it ourselves if we can run it. Here we do not expect to beat FigEx. The test is how much of the gap above YOLO11 a small detector plus snapping closes, per unit of compute.

Ablations. Each isolates one ingredient:

  • snapping on raw gradient magnitude instead of $E$;
  • plain local binary patterns instead of the ternary code;
  • the texture term alone and the gain term alone;
  • seam channels as input without snapping, and snapping without seam channels.

Why not just…?

Why not just add photometric augmentation to the SimCFS simulator and let the CNN learn invariance? This is the most serious threat, and it may well match us on in-distribution data. That is why every arm, including the SimCFS baseline, gets the same augmentation. The stress test settles the question. Learned invariance should degrade once test perturbations leave the augmented range, while the ternary code is invariant to any multiplicative gain by construction. If the augmented baseline matches our full system even under out-of-range perturbations, we drop the descriptor claim. The surviving contribution would then be one-dimensional snapping, and the gradient-snapping ablation tells us whether it survives on its own.

Why not just use FigEx or a bigger vision-language model? FigEx's gains come largely from reading the caption, which our method does not attempt. Its MedICaT AP of 0.203 shows that even a 7B model struggles to localize panels. Our proposal targets corpus-scale cropping where a 7B forward pass per figure is the cost to avoid. Our claim is about accuracy per unit of compute and about tighter boxes at strict thresholds, not about matching caption alignment.

Why not just use classical gutter and line detection, or plain local binary patterns? The NLM pipeline needed manual stitched-versus-gutter sorting and was confused by axes and cell membranes. Liao et al. show that plain local binary patterns flip under noise in flat regions and trail the ternary code by more than 10% F-score. Both remain in the study as ablations rather than being dismissed on citation alone.

Why not refine boxes with iterative optimization such as active contours? For axis-aligned rectangles, the exact maximizer of a separable edge score is a one-dimensional arg-max per edge. An iterative loop would add latency without finding a better solution under the same score. We will time it against snapping as a check.

Proposed Experiments

Main comparison. The main table covers Glomeruli-2000, ImageCLEF 2016, and BioSci-Fig. It reports mAP at IoU 0.5, 0.75, and 0.9, plus a bleed measure: the mean fraction of predicted-box area lying outside the matched ground-truth box. We also report ImageCLEF 2015 accuracy against the 84.64% NLM result, and MedICaT AP against FigEx's 0.203.

Downstream check of crop quality. Yao et al. show that SimSiam pretraining on SimCFS-mined Glomeruli crops reaches F1 0.900, versus 0.845 from scratch. We will repeat that protocol with crops from each arm. If tighter boxes matter, snapped crops should exceed 0.900.

Speed check. We measure latency per figure on CPU and GPU for seam-map construction, snapping, and the full pipeline, and set it beside FigEx where it can be run.

Decisive outcomes. The idea succeeds if three things hold:

  • the full system beats reproduced SimCFS at [email protected] and @0.9 on Glomeruli-2000;
  • its accuracy drop under out-of-range photometric perturbation is clearly smaller than that of the augmentation-trained baseline;
  • the added cost is a small fraction of detector inference.

The idea fails if the augmented baseline matches it under perturbation, or if gradient-based snapping performs as well as seam-evidence snapping.