Acquisition-Consistent Deformable Sampling for Weakly Supervised Optical Biopsy

Optical biopsy microscopes image fresh, uncut tissue during surgery and return cell-level pictures within seconds to minutes. A model that reads those pictures and marks where tumor is would help surgeons decide whether to cut more. Two practical problems stand in the way. First, experts rarely have time to draw tumor outlines, so models are trained only on coarse labels such as "this image is diagnostic" or "this patient has a glioma." Second, nearly every dataset comes from a single hospital, and the models break when the microscope, magnification, probe handling or patient population changes. This note proposes one mechanism aimed at both problems. We use deformable convolution, a convolution layer that learns where to look by shifting its sampling points off the fixed square grid. We add a label-free rule: when the image is geometrically distorted in the ways another site's microscope would distort it, the learned sampling points must move with the tissue. The rule supplies the training signal that coarse labels cannot give the sampling points, and it ties that signal to the kind of geometric variation that separates one site's optics from another's.

The core claim is narrow and testable. Simply adding deformable layers to a weakly supervised model is unlikely to help, and may hurt, because nothing stops the sampling points from drifting toward whatever background cue predicts the image label. Constraining the sampling points to follow acquisition transforms should do three things. It should stabilize them under weak labels. It should produce activation maps that follow irregular infiltrative borders more closely than rigid grids do. It should shrink the performance drop when a model trained at one site is applied to another, and it should beat a rigid network given exactly the same geometric augmentation.

Background and the gap

Weakly supervised localization already exists for optical biopsy. Weakly-Supervised Learning-Based Feature Localization for Confocal Laser Endomicroscopy Glioma Images trains on image-level diagnostic tags from 6,287 confocal laser endomicroscopy (CLE) images of 20 patients. CLE is a fiber-probe microscope that images living tissue at cellular resolution. The paper builds multi-scale class activation maps, which are heatmaps of where a classifier's evidence lies, and adds hand-designed inhibition and integration steps across layers. It reports 86% and 88% mean IoU against 310 neurosurgeon-annotated images. It uses rigid convolutions and a single center, and it never tests transfer. This paper motivates our work and supplies the strongest localization baseline.

Cross-site fragility in CLE is documented by Patch-based Carcinoma Detection on Confocal Laser Endomicroscopy Images - A Cross-Site Robustness Assessment. That study found that a patch classifier trained on oral cavity images reaches 89.45% accuracy and 0.955 AUC on vocal cord images, but the reverse direction drops to 68.53%. It used patch-level labels throughout. Whether that asymmetry survives, or worsens, under weak labels is unknown.

For stimulated Raman histology, a label-free chemical-contrast microscope, OpenSRH provides a public single-center benchmark of 307 patients. It reports that a ResNet50 trained with supervised contrastive learning on patient-level diagnoses reaches 87.5% patch and 90.0% patient accuracy, and that it beats a small vision transformer. That last result constrains us: at this data scale, keeping a convolutional inductive bias is the right default, and an attention-only model is the weaker bet.

Foundation models for fast, label-free detection of glioma infiltration shows the large-compute alternative. FastGlioma was pretrained on about 4 million images and reaches 92.1 ± 0.9% AUC on an international prospective cohort. It outputs a slide-level infiltration score rather than a boundary, and it depends on data volumes that newer modalities lack.

The method we reuse comes from Deformable Convolutional Networks and Deformable ConvNets V2. In these works, a small parallel layer predicts a two-dimensional offset for each kernel tap, and the features at those fractional positions are read by bilinear interpolation. In medicine, DUNet showed that deformable kernels follow tortuous retinal vessels and reported 0.9697 accuracy and 0.9856 AUC on DRIVE. DUNet trained on dense pixel masks. The research file itself flags that offset predictors trained on small, noisy, weakly labeled data tend to collapse or wander. That warning is the problem our constraint targets.

The closest adaptive-kernel approach to domain shift, Domain and Content Adaptive Convolution for Domain Generalization in Medical Image Segmentation, adapts kernel weights to the input domain's style. It is fully supervised and radiological. We instead adapt where kernels sample, and we constrain that adaptation without labels.

Core mechanism

Let $x$ be an image and $T$ a randomly drawn acquisition transform. The transforms are chosen to mimic what differs between optical biopsy setups: isotropic rescaling for magnification and pixel-pitch differences, shear and anisotropic stretch for probe sweep and scan motion, small rotations, and smooth local warps for tissue compression. Write $J_T(p)$ for the local Jacobian of $T$ at location $p$. At a deformable layer, the effective sampling position of kernel tap $n$ around location $p_0$ is $s_n(x, p_0) = p_n + \Delta p_n(x, p_0)$. Here $p_n$ is the fixed grid offset and $\Delta p_n$ is the learned offset. We add an offset-consistency loss

$$ \mathcal{L}_{\text{oc}} = \mathbb{E}_{T,\,p_0,\,n}\big\| s_n(Tx,\, T p_0) - J_T(p_0)\, s_n(x, p_0) \big\|^2 , $$

which asks the sampling pattern on the transformed image to be the transformed sampling pattern from the original. The total objective is the weak-label loss plus $\lambda \mathcal{L}_{\text{oc}}$. The weak-label loss is image-level cross-entropy for CLE, or patient-level supervised contrastive loss for SRH as in OpenSRH. We also add a feature-agreement term at corresponding locations so that the constraint has a downstream effect.

The reasoning is as follows. If the sampling points transform with the tissue, then the features they read are, to first order, unchanged by $T$. That gives geometric invariance through the mechanism itself, rather than through the extra capacity a rigid network spends memorizing augmented copies. The same loss also anchors the offsets to tissue-relative positions, so they cannot drift freely toward label-correlated background.

Two engineering choices keep this compatible with intraoperative use. Deformable layers go only in the last two backbone stages, as DCNv2 does. We also measure per-image latency against the seconds-scale budget, because Algorithm-hardware Co-design for Deformable Convolution documents the memory-access cost of irregular sampling. For localization, we project class evidence back onto the actual deformed sampling positions rather than onto the rigid feature grid. This gives a sub-patch map that can follow curved borders, and the mapping is cheap to compute.

Research program

The work has three parts. First, we build the model on a ResNet50 backbone so that it matches the strongest OpenSRH baseline architecture, and we train it under weak labels only. Second, we analyze the offsets directly. We record how far sampling points move, whether they collapse onto a single location, what fraction land on tissue versus probe border or background, and how consistent they are across repeated acquisitions. This tests whether the regularizer does what we claim, and the same analysis applied to an unconstrained deformable network tests the instability warning. Third, we test transfer under weak supervision. This directly asks the open question from the cross-site study: does the "superset" advantage, and the collapse in the reverse direction, persist when patch labels are withheld during training?

Baselines

The naive reference point is a rigid ResNet50 classifier trained on the same weak labels, with standard flips and crops, and read out through ordinary class activation maps. Every other number is measured relative to it.

The strongest existing localization method is the multi-scale activation-map pipeline with lateral inhibition and collateral integration from Weakly-Supervised Learning-Based Feature Localization for Confocal Laser Endomicroscopy Glioma Images. It reports 86% mean IoU on intermediate features and 88% on restrictive fine features over the 310 annotated images. We will reimplement it on our backbone with the same training images and labels. Our method also composes with its inhibition and integration steps, so we will report both a head-to-head comparison and the combination. Beating it means a higher IoU on the same 310 images, with the gain concentrated in boundary-sensitive measures such as boundary F-score.

For SRH classification, the strongest reported number is ResNet50 with supervised contrastive learning from OpenSRH, at 87.5% patch-level and 90.0% patient-level accuracy. We will rerun it on the official splits with an identical tuning budget. For cross-site CLE, the reference is the patch-supervised result of 89.45% accuracy and 0.955 AUC for oral cavity to vocal cords, and 68.53% for the reverse, from the cross-site robustness study. A weakly supervised model is not expected to reach the fully supervised number in the easy direction. A clear win would be narrowing the gap in the vocal-cord-to-oral-cavity direction relative to a weakly supervised rigid model. For the adaptive-kernel family, we run Domain and Content Adaptive Convolution with its segmentation head replaced by our weak-label head.

The ablation that isolates the key ingredient has four arms, all sharing the same transform distribution and compute. The first is a rigid network with the acquisition transforms used purely as augmentation. The second is a rigid network with the same transforms and a feature-consistency loss. The third is an unconstrained DCNv2 network with the same augmentation. The fourth is DCNv2 with the offset-consistency loss. Only the difference between the third and fourth arms, and between the first two arms and the fourth, says whether constraining where the kernels sample matters. All arms get the same number of epochs, the same hyperparameter search budget of about twenty trials on a source-site validation split, and no access to target-site data during tuning.

Why not just…?

Why not just train a rigid CNN with heavy geometric augmentation? This is the most serious threat. If the gains come from the transform distribution rather than from the deformable mechanism, the rigid-plus-augmentation arm will match ours. DUNet's advantage over augmented U-Nets (DUNet) was measured under dense supervision, so it does not settle the weak-label case. The four-arm ablation above settles it. If the rigid arm with augmentation and feature consistency is within noise of our model on both the cross-site test and the IoU test, we drop the deformable claim and report the transform distribution as the contribution.

Why not just add deformable layers and let them learn? Under coarse labels, nothing tells the offsets where the lesion is, and the research file's own summary of the deformable literature names collapse and drift as the failure mode. We expect unconstrained DCNv2 to look fine on the source site and degrade on the target site. The offset diagnostics, especially the fraction of sampling points landing on probe border or background, will show whether this happens.

Why not just resample every image to a common pixel size and normalize intensities? Resampling with known metadata removes global magnification differences, and we include a resampled, intensity-normalized rigid baseline for exactly that reason. It cannot correct local shear from probe motion or tissue compression. Moreover, the vocal-cord-to-oral-cavity drop in the cross-site study arose within one device family, which points to morphology and artifact differences rather than global scale or color. Stain-style normalization such as Adaptive Stain Normalization for Cross-Domain Medical Histology targets chemical color variation, and CLE and SRH contrast is not produced by stains. If resampling alone closes most of the gap, our claim shrinks to the local-distortion residual, and we will say so.

Why not just use a large pretrained foundation model? FastGlioma's 92.1% AUC across centers (FastGlioma) shows that scale works when millions of in-modality images exist. For CLE glioma, oral cavity and vocal cord data, cohorts number in the tens of patients, and OpenSRH reports a domain gap from ImageNet pretraining. FastGlioma also outputs a score rather than a boundary. Where its weights or a comparable encoder are accessible, we will fine-tune it as an additional baseline. Our method can also sit on top of such an encoder's later stages, so the two are complementary rather than exclusive.

Proposed Experiments

On the CLE glioma data, we train with image-level diagnostic tags only. We report IoU and boundary F-score on the 310 neurosurgeon-annotated images, separately for intermediate and fine features, and we compare against the 86% and 88% IoU reported for the existing pipeline.

On the oral cavity and vocal cord CLE datasets, we discard the patch labels for training and keep them for evaluation. We train with image-level or patient-level tags under leave-one-patient-out validation and test in both transfer directions. This yields accuracy and AUC, plus patch-level localization scored against the withheld patch labels. Both CLE corpora come from the cited papers, and their public availability is not stated in our sources. We will request access, and if it is refused, the cross-anatomical-site test reduces to OpenSRH plus partner data.

On OpenSRH, we train with patient-level supervised contrastive learning and report patch accuracy, patient accuracy and mean class accuracy on the official splits. Because OpenSRH is single-center, we add two shift tests. The first is a held-out geometric shift that uses transform magnitudes outside the training range. The second, if a partner site can contribute even a few dozen SRH cases, is a true external-center evaluation. We report the source-to-target gap rather than raw target accuracy. Latency per field of view on a single clinical-grade GPU is reported for every arm.

Decisive checks

The proposal stands or falls on three outcomes. First, on the weakly supervised vocal-cord-to-oral-cavity transfer, the offset-constrained model must beat the rigid-plus-augmentation-plus-consistency arm by more than the seed-to-seed spread across five seeds. Second, localization IoU on the 310 annotated CLE images must match or exceed the existing pipeline's 86% and 88%, with the improvement showing up in boundary measures. Third, the offset diagnostics must show that unconstrained deformable layers drift onto non-tissue regions under weak labels and that the constraint prevents it. If the first check fails while the third succeeds, we have a useful negative result: stabilizing offsets does not buy cross-site robustness beyond what augmentation already provides. We would publish that, and it would steer the field toward scale or data collection instead.