In endoscopic submucosal dissection (ESD), an endoscopist removes an early tumor from the inner lining of the gut by cutting through the layer just beneath it. Before cutting, they inject fluid into that layer, the submucosa, to lift the tumor away from the muscle wall. This fluid "cushion" is the only thing standing between an electrocautery knife and a perforation. The cushion does not stay put. It is absorbed by surrounding tissue, and it leaks out through every incision. Endoscopists judge how much cushion remains by eye through a single camera, and they re-inject by feel. Current AI systems for ESD do not track the cushion at all. They recognize which procedural phase is underway, or they draw a safe-cutting region on the visible tissue surface.
We propose to estimate and forecast the cushion's thickness during the procedure as a hidden physical quantity. The estimator combines three sources of information. The first is a log of every injection, with volume, fluid type and the back-pressure at the needle. The second is the endoscopic video, which shows the cushion bulge and each new incision. The third is a handful of patient and lesion facts, such as organ site and suspected fibrosis (scar tissue that resists lifting). The estimator is a filter built around a simple model of fluid entering and leaving the cushion. Injections feed the model. The video corrects its estimate whenever the bulge is visible and tells it when leak paths open. The patient facts set its starting absorption and leak rates. The output is a cushion-thickness estimate with uncertainty, plus a warning of how many seconds remain before the margin drops below a chosen safety threshold.
Core Claim
The safety margin in ESD is a three-dimensional, time-varying quantity: fluid volume under the lesion. It cannot be read reliably from surface pixels alone, and it should not be learned as an unstructured black box from a few dozen procedures. The claim is that a physically structured state estimator outperforms two alternatives at predicting cushion thickness and time-to-collapse, especially late in the dissection when the flap hides the cushion from the camera. The estimator treats injection telemetry as known inputs, video as noisy observations and incision events, and tabular facts as priors. The first alternative is vision-only temporal models. The second is generic learned fusion of video with tabular data.
This reframes "heterogeneous data fusion" for this setting. The different modalities are not features to be mixed. They play different causal roles in a single dynamical system. A thin injection log carries information about inputs to the system that no amount of video encodes. Feeding that log to a network as an extra feature vector invites the known failure in which the high-dimensional stream drowns out the low-dimensional one.
Research Program
The prior work splits cleanly along the two directions. The surgical literature motivates the problem but offers no quantitative handle on it. Yu Xia et al. lift a roughly 60 × 25 mm esophageal lesion with saline cushions. Cheng, Yip et al. mix in hyaluronic acid to keep a duodenal cushion stable longer. Ping Ma et al. tunnel through the submucosa to remove masses larger than 8 cm. In all three, maintaining the cushion is empirical. Teoh et al. supply our measurement method. They show that endoscopic ultrasound (EUS) resolves the wall layers of porcine bladders after saline injection, so ultrasound can serve as ground truth for cushion thickness.
The vision literature constrains what is already solved. AI-Endo and SPRMamba recognize ESD phases from video in real time. PDZSeg segments the dissection zone. ETSM regresses a two-dimensional confidence map of the safe cutting plane. All four treat injection as an event label or ignore it, and none estimates depth. We reuse their video encoders and their phase labels as tools, not as competitors on their own tasks.
The fusion literature supplies both warnings and a strong generic baseline. Kocielnik et al. find that one modality suppresses the others under joint training in robotic surgery, and they fix this by training each modality's encoder separately first. TabMixer conditions spatial, temporal and channel mixing of video on tabular embeddings to estimate a hidden physical pressure from cardiac MRI. That is the closest analogue to our task in any field. TFormer reports that adding a modality can hurt on fewer than 500 training cases. Our cohorts will be that small, which argues for physical structure over capacity.
The work has three parts.
Instrumented data collection. We fit a standard endoscopic injection needle to a syringe pump with a logging pressure transducer, which records volume, flow and back-pressure at high rate. Ex vivo porcine stomach and esophagus specimens give dense ground truth. A high-frequency ultrasound probe placed under the specimen measures cushion thickness continuously beneath the lesion. Absorption is weak ex vivo, so these sessions mainly expose the incision-leak dynamics. In vivo porcine ESD, of the kind AI-Endo already performed in live animals, supplies absorption dynamics. There we take sparse EUS miniprobe measurements at scheduled pauses and immediately before each re-injection. Each session crosses fluid type (saline versus a viscous agent) with lesion size and site, and a subset includes induced fibrosis.
The estimator. The hidden state is the cushion volume $V(t)$ beneath the lesion, which maps to thickness $h(t)$ through a per-site area estimate. Its dynamics are
$$\dot V(t) = u(t) - k_a V(t) - k_\ell\, \ell(t)\, g(V(t)),$$
where $u(t)$ is the logged injection flow, $k_a$ is the absorption rate, $\ell(t)$ is the cumulative open incision length detected from video, $g$ is a monotone pressure-driven leak term, and $k_\ell$ scales the leak. A small network maps the patient facts, the fluid type and the back-pressure profile of the first injection to priors on $k_a$ and $k_\ell$. Back-pressure is the one direct mechanical probe of tissue compliance we get.
A video encoder serves two purposes. Its incision detector updates $\ell(t)$. It also produces a noisy reading of bulge height, together with a learned reliability score. That score falls toward zero when the flap hides the cushion, or when smoke or blood obscures the view. An extended Kalman filter, a standard recursive estimator that blends a dynamics prediction with uncertain measurements, combines everything. The filter is trained end to end on the ultrasound labels, and the encoders are pretrained separately in the staged manner that Kocielnik et al. recommend.
Forecasting and alarms. Rolling the dynamics forward from the current estimate gives a predicted time until $h$ falls below a threshold $h^*$. We report results across a range of $h^*$ values chosen with endoscopists, rather than committing to one value.
Baselines
The naive reference point is a fixed re-injection clock. It predicts that the cushion decays from its post-injection height at a single population-average rate, ignores video and patient facts, and is fit on the training sessions. Any proposed method must beat it clearly on time-to-threshold error, or the problem is not worth the machinery.
The strongest vision-only competitor is a temporal model built from the published ESD encoders. We take the frame encoder of AI-Endo or the DINOv2-with-LoRA backbone of ETSM, add the temporal head of SPRMamba, and train it to regress $h(t)$ directly from video on our labels. It is the strongest vision competitor because it is exactly what the field would build next. Published reference numbers exist only for neighbouring tasks: ETSM reports a confidence-map MAE of 3.18, AI-Endo reports 91.04% phase accuracy on its development data, and SPRMamba reports 87.64% on ESD385. There is no reported result on Lacuna for cushion-thickness estimation, so we establish the vision-only number ourselves.
The strongest generic fusion competitor is TabMixer. We condition its video mixing on the tabular facts plus a summary of the injection log. TabMixer reports a mean absolute error of 6.66 mmHg for pulmonary artery pressure, a different task and unit, so it shows that the architecture can recover hidden physical state but gives no target to beat here. We also run a concatenation-fusion model trained in stages.
Every model receives the same sessions, the same pretrained encoders, the same patient-level cross-validation splits and the same hyperparameter search budget. The proposal counts as winning only if two conditions hold together. First, its mean absolute error on thickness must be lower than TabMixer's in the occluded late-dissection segments, with confidence intervals that do not overlap across held-out animals. Second, at matched false-alarm rates, its alarm lead time must be longer.
Three ablations isolate the key ingredient. One replaces the physical state with a free learned latent of the same size, a plain learned state-space sequence model. One removes the injection telemetry. One removes the video reliability score.
Why not just…?
Why not just regress cushion height from video with a strong temporal vision model? This is the real threat. Early in a dissection the bulge is plainly visible, and vision may suffice there. The argument against it is specific to late dissection. Once the flap folds over, the camera views the cushion edge-on or not at all, and smoke and blood degrade the view, failure modes that PDZSeg and ETSM themselves document. We stratify every result by visibility, using an annotated flap-occlusion label. If vision-only matches our estimator even in occluded segments, we drop the physical model and report that video suffices. That would itself be a useful negative result.
Why not just a decay clock based on volume and fluid type? It may come close for intact cushions. However, leak through incisions is driven by events that only the video sees, and absorption varies with the tissue. The experiment that settles the point compares the clock and our filter on the segments that follow each new incision. If the clock does as well there, the video's contribution is nil.
Why not just feed everything to TabMixer or a cross-attention fusion network? With tens of procedures, a free network must learn conservation of fluid from data. TFormer's small-cohort degradation and Kocielnik's suppression result both suggest it will underuse the sparse injection log. The free-latent ablation tests exactly this question at equal capacity.
Why not just use a longer-lasting viscous agent? Hyaluronic-acid cushions still dissipate and leak during large resections, as the Xia and Ma cases illustrate. Fluid type is a crossed factor in our design, so we measure directly how much a viscous agent narrows the forecasting problem rather than assuming it removes the problem.
Proposed Experiments
The primary benchmark is the new instrumented porcine set, split by animal. The main metrics are thickness mean absolute error in millimetres, error in forecasting time-to-threshold, and alarm sensitivity at fixed false-alarm rates with lead time. We also check calibration, asking whether the stated uncertainty covers the ultrasound value at its nominal rate.
Existing benchmarks serve as checks rather than targets. The incision and injection event detector is pretrained on the AI-Endo and ESD385 phase labels. We confirm that the vision backbone still matches the published phase-recognition accuracy, so that any gain is not explained by a weaker encoder.
Three tests are decisive. The first is the occluded-segment comparison against vision-only and TabMixer. The second is a fluid-type transfer test: train on saline sessions, test on viscous-agent sessions, where a physical model can adapt through $k_a$ while black-box models must extrapolate. The third is the post-incision test against the decay clock. If the filter wins the first two and the clock loses the third, the claim stands. If vision-only wins the first, the claim fails.
References
- Endoscopic submucosal dissection for resection of giant adenoid cystic carcinoma of the esophagus
- Endoscopic removal of a submucosal embedded foreign body in the duodenum
- A case report of piecemeal submucosal tunnel endoscopic resection for a giant esophageal leiomyoma larger than 8 cm
- A proof-of-concept study on endoscopic ultrasound-guided biopsy of detrusor muscle in porcine bladders
- Intelligent surgical workflow recognition for endoscopic submucosal dissection with real-time animal study
- SPRMamba: Surgical Phase Recognition for Endoscopic Submucosal Dissection with Mamba
- PDZSeg: adapting the foundation model for dissection zone segmentation with visual prompts in robot-assisted endoscopic submucosal dissection
- ETSM: Automating Dissection Trajectory Suggestion and Confidence Map-Based Safety Margin Prediction for Robot-Assisted Endoscopic Submucosal Dissection
- Deep Multimodal Fusion for Surgical Feedback Classification
- TabMixer: Noninvasive Estimation of the Mean Pulmonary Artery Pressure via Imaging and Tabular Data Mixing
- TFormer: A throughout fusion transformer for multi-modal skin lesion diagnosis