Surgical papers often show a deformation as a compound figure, meaning one image file containing several sub-panels: the liver before and after insufflation, tissue before and after retraction, or a cut photographed at four moments. These panels are rare examples of the same tissue under known manipulations, and they come with captions that say what was done. Current literature-mining pipelines split such figures with an object detector, crop each panel, and treat the crops as unrelated images. If someone then tries to estimate how the tissue moved between panels, every pixel of detector error enters the deformation estimate. A crop that bleeds into the white gutter, a neighbor panel, or the printed "B" label does not just add noise. Panel labels and gutters sit in the same place in every panel of a grid, so a registration method sees them as stable structure, pulls the border region toward zero motion, and produces false stretching where real tissue meets the misplaced edge.
We propose to estimate panel boundaries and inter-panel tissue deformation together. The registration residual tells the decomposer where a boundary is wrong, and the deformation is computed only over pixels the decomposer is confident belong to tissue. The two-sentence version: decomposition errors in compound figures turn into fake tissue deformation in the same way interpolation errors turn into fake volume change in medical registration. Fitting the boxes and the warp jointly, with a tissue-plausibility prior on the warp, removes that artifact and produces a clean corpus of text-annotated deformation pairs.
Core Claim
Treating panel detection and panel-to-panel registration as a cascade adds a structured, measurable bias to the recovered deformation. A joint fit removes most of that bias without lowering detection accuracy on ordinary compound figures. The bias can be measured with the idea behind Interpolation Artefacts in Non-rigid Registration. That paper registered same-day repeat MRI scans of unchanged subjects and showed that linear interpolation alone produced a −0.22% "fake" volume change, which Cardinal splines halved to −0.11%. Our counterpart is the zero-deformation compound figure: two panels cropped from the same video frame, or from a frame and a copy warped by a known amount. Any deformation recovered beyond the known one is an artifact of the pipeline.
This builds on, and differs from, three groups of prior work. The literature-mining pipelines supply the decomposition machinery and motivate the problem. PMC-CLIP used a DETR detector (a transformer that predicts object boxes directly) to split PubMed Central figures into 1.6M sub-panels, with 73% subcaption alignment accuracy. Open-PMC showed that training on decomposed sub-panels instead of whole compound figures gains about 10 F1 points in pneumonia detection, and it reports that its decomposer struggles on dense photographic series. COVID-19-CT-CXR treats sub-panels as independent static images. All three treat panels as exchangeable and never ask whether two panels show the same tissue. That relation is our main signal.
SimCFS supplies two techniques we reuse. The first is synthesizing compound figures from single images. The second is Side Loss, a boundary penalty that shrink-wraps boxes and reached 88.9% [email protected] on ImageCLEF 2016. Side Loss is also the strongest competitor. It makes boxes tighter, but tightness is judged against an annotated box, not against whether the crop supports a correct correspondence.
The cascaded-restoration papers motivate the joint formulation. GFN showed that chaining deblurring and super-resolution magnifies ringing, and UniINR beat a chained rolling-shutter-plus-interpolation pipeline by 8.49 dB. Both remove the cascade inside a single image-formation problem. We instead couple a discrete layout decision with a continuous geometric one.
Finally, the surgical deformation work supplies the deformation prior and the test imagery. Land-Reg regularizes liver deformation with a local-isometry term, which keeps distances along the tissue surface roughly constant. It also documents the cascade failure we generalize: a wrong rigid stage forces the non-rigid stage to bend the organ to compensate. Forplane, Endo-4DGS and HFGS reconstruct deforming tissue from continuous video. We do not compete with them. We use their benchmark videos as a source of panels with known deformation.
Research Program
The system takes a compound figure and its caption. It starts from a SimCFS-style detector. A light classifier, trained on caption cues ("before/after", "(a)–(d)", "following retraction") and on visual similarity of the crops, proposes which panel pairs depict the same tissue. For each proposed pair $(i,j)$ we solve
$$\min_{b_i,b_j,\phi}\; \sum_{x \in \Omega(b_j)} w(x)\,\rho\big(I_j(x) - I_i(\phi(x))\big) \;+\; \lambda\, R_{\text{iso}}(\phi) \;-\; \mu \sum_{k\in\{i,j\}} \log p_{\text{det}}(b_k).$$
Here $b_i, b_j$ are the panel boxes and $\Omega(b_j)$ is the pixel region inside box $b_j$. The warp $\phi$ maps panel $j$ coordinates to panel $i$, $I_i, I_j$ are the panel intensities, and $\rho$ is a robust photometric penalty (Charbonnier). The weight $w(x)$ is a learned tissue-versus-layout mask that down-weights gutters, letters and scale bars. $R_{\text{iso}}$ is the local-isometry penalty adapted from Land-Reg to 2D, and $p_{\text{det}}$ is the detector's box likelihood, so the boxes cannot drift arbitrarily. We alternate between gradient steps on $\phi$, initialized from a pretrained optical-flow model as HFGS does, and box refinement over a small neighborhood of the detector's proposal. A box edge moves when moving it lowers the registration energy by more than it lowers detector likelihood. Panels the pairing classifier marks as unrelated fall back to the detector alone, so ordinary figures are unaffected.
The training and evaluation data come from simulated compound figures with exact ground truth. We assemble frames from EndoNeRF, StereoMIS and SCARED into grids following the SimCFS-AUG recipe. The grids include gutters, panel letters, scale bars and the irregular, tightly packed layouts the file flags as hard. Pairs come in three kinds. Same-frame pairs have true deformation of zero. Synthetic-warp pairs apply a known smooth warp. Real temporal pairs take frames seconds apart, with pseudo-ground-truth motion from a fitted 4D Gaussian splatting reconstruction. We then mine real surgical compound figures from PubMed Central with the Open-PMC pipeline and hand-annotate a held-out set with boxes and sparse landmark correspondences. The released output is a corpus of panel pairs with deformation fields and the caption phrase describing the manipulation.
Baselines
The naive reference point is registration of uncropped or uniformly gridded panels: split the figure at fixed fractions and register the halves with the same optical-flow initialization and isometry prior. This shows how much any decomposition matters.
The main cascade baseline is the SimCFS detector with Side Loss, trained on the same simulated surgical grids. Its crops go to an identical registration module with the same robust penalty, the same isometry weight, the same flow initialization and a matched hyperparameter search budget. SimCFS is the strongest competitor because Side Loss targets exactly the bleed we blame for fake deformation. Its reported 88.9% [email protected] on ImageCLEF 2016 is the detection number we must not fall below on that benchmark. We will retrain from the public code and report our reproduced value alongside it. A DETR detector configured as in PMC-CLIP and Open-PMC is a second cascade baseline, since that is what large-scale mining actually uses.
For the deformation side, no reported result was found on Lacuna for registration accuracy between compound-figure panels. Every deformation number will therefore come from our own controlled runs.
Two ablations isolate the key ingredient. The first keeps the learned layout mask $w$ but freezes the boxes, which tests whether masking alone explains the gain. The second keeps joint box refinement but drops the isometry prior, which tests whether the prior is what makes registration residuals a trustworthy signal for moving boxes.
We will count the joint method as better only if all three of the following hold on held-out simulated layouts and on the annotated real figures. It must cut zero-deformation fake strain and endpoint error relative to Side-Loss-plus-registration by a margin larger than the seed-to-seed spread. It must reduce border-region error specifically. It must leave ImageCLEF 2016 [email protected] within noise of the retrained SimCFS.
Why not just…?
Why not just use Side Loss boxes, which are already tight? Side Loss penalizes predicted edges that extend past the annotated box. It has no notion of whether a crop edge falls on tissue that has a match in the partner panel. When it over-shrinks, it cuts away tissue, and the warp has to extrapolate. When it under-shrinks by a few pixels, the gutter content stays in, and that is exactly the content that biases registration. This is a real threat in easy layouts. The border-band analysis in the experiments will show where, if anywhere, our method helps beyond it, broken down by gutter width and by how dense the layout is.
Why not just use a robust registration loss and ignore the boxes? Robust penalties down-weight pixels that disagree. In a grid layout, however, letters, gutters and scale bars agree across panels, because they sit at the same relative positions. A robust loss therefore trusts them and drags the border toward zero motion. That argument could be wrong. The frozen-box ablation, which includes the learned mask, is the direct test. If it matches the full joint method, we will report that the mask is sufficient and drop box refinement from the proposal.
Why not just prompt a large vision-language model with the whole figure and ask for panel boxes and correspondences? Open-PMC already uses GPT-4o for caption alignment. It and PMC-CLIP both report that monolithic figures confuse panel-level semantics, and PMC-CLIP's alignment still fails in 27% of cases. Neither paper tests pixel-level correspondence, which is what deformation needs. We will run a GPT-4o box-and-landmark prompt on the annotated real set as an extra comparison under the same evaluation.
Why not just learn deformation priors from surgical video and skip the literature? For deformation modeling alone, that is the better source, and EndoNeRF-style videos are denser than any figure. What the literature adds is breadth: organs, procedures and manipulations absent from the six-clip-scale video benchmarks, each paired with text naming the manipulation. Our central claim, though, concerns error propagation inside literature mining, and that claim stands even if the corpus turns out to be of modest use for downstream modeling. We present it that way.
Proposed Experiments
The first decisive check is the zero-deformation test. On same-frame panel pairs drawn from EndoNeRF, StereoMIS and SCARED, we compare mean spurious strain and endpoint error in pixels between the cascade baselines and the joint method. We stratify by distance from the crop border, expecting the cascade's error to concentrate in a band a few gutter-widths wide. We then repeat the test on synthetic-warp pairs at increasing warp magnitude. The aim is to confirm that the joint method does not suppress real motion near borders while removing fake motion.
The second check is real temporal pairs, scored against pseudo-ground-truth motion from a reconstruction in the style of HFGS. We report endpoint error and the fraction of pairs where joint box refinement moved an edge. On the hand-annotated PubMed Central surgical figures, we measure box [email protected], landmark transfer error and accuracy at recovering panel order from estimated deformation magnitude.
The third check is the no-harm test on ImageCLEF 2016, where most figures are unrelated panels. Retrained SimCFS, with its reported 88.9% [email protected], is the bar. A drop beyond noise would mean the pairing classifier is wrongly coupling unrelated panels, and that classifier would need to be fixed before any corpus is released.
Finally, we check whether caption phrases predict deformation statistics. Pairs captioned "retraction" should show larger, more anisotropic strain than pairs captioned "insufflation". This tests whether the mined corpus carries usable text–deformation signal beyond the artifact-removal result.
References
- Interpolation Artefacts in Non-rigid Registration
- PMC-CLIP: Contrastive Language-Image Pre-training Using Biomedical Documents
- Advancing Medical Representation Learning Through High-Quality Data
- COVID-19-CT-CXR: A Freely Accessible and Weakly Labeled Chest X-Ray and CT Image Collection on COVID-19 From Biomedical Literature
- Compound Figure Separation of Biomedical Images with Side Loss
- Gated Fusion Network for Joint Image Deblurring and Super-Resolution
- UniINR: Event-Guided Unified Rolling Shutter Correction, Deblurring, and Interpolation
- Preoperative-to-intraoperative Liver Registration for Laparoscopic Surgery via Latent-Grounded Correspondence Constraints
- Efficient Deformable Tissue Reconstruction via Orthogonal Neural Plane
- Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting
- HFGS: 4D Gaussian Splatting with Emphasis on Spatial and Temporal High-Frequency Components for Endoscopic Scene Reconstruction