Which Frames to Afford: Energy-Causal Burst Sampling for Deformable Tissue Reconstruction

A wireless capsule endoscope, or any untethered micro-device inside the body, runs on a tiny energy store that refills irregularly. It refills from an external inductive field, from radio waves, or possibly from the body's own motion. Such a device obeys energy causality: it can only spend energy it has already collected, and anything harvested while its small battery is full is lost. The methods that reconstruct moving soft tissue in 3D over time, notably 4D Gaussian splatting (a scene made of many small 3D blobs whose positions are moved over time by a learned deformation function), all assume an unbroken 30-frame-per-second video stream fed to a desktop GPU. This proposal asks a question that sits between those two facts. If the device can afford to wake its camera, light, and radio for only a small fraction of frames, which frames should it send to an external receiver so that the receiver's 4D tissue model stays accurate?

Our answer has two parts. Frames are worth most in short contiguous bursts placed on deformation events, because deformation models and point trackers need nearby frames to establish correspondence. Every wake-up also pays a fixed overhead before a single useful pixel is produced. Together these make the energy-optimal sampling pattern bursty, in exactly the sense that energy-harvesting theory predicts for transmitters with a processing cost. We build an online scheduler that respects energy causality. It estimates the value of a burst from cheap on-board signals and decides when to wake. We test it by replaying real endoscopic video under simulated harvesting traces and measuring how well a 4D reconstruction built only from the chosen frames recovers all the frames it never saw.

Core Claim

Under a fixed per-wake-up overhead and energy causality, the frame-selection policy that minimizes reconstruction error of a deforming tissue model is a thresholded burst policy. It stays asleep through quiescent tissue drift and fires a short contiguous burst when predicted deformation is high and the stored energy clears a level that rises over time. When harvested energy arrives correlated with tissue motion, the policy must also hold a reserve before the event. Spending energy as it arrives then systematically misses the very contractions that produced it. We claim this policy, run causally with a learned value estimate, beats uniform duty-cycling, greedy energy-neutral capture, and motion-triggered capture at matched energy. We claim the margin is largest on high-deformation segments and under motion-correlated harvesting.

The scheduling structure comes from the energy-harvesting literature. Throughput Maximization for an Energy Harvesting Communication System with Processing Cost shows that a nonzero circuit cost $\epsilon$ turns continuous low-power transmission into bursts, via a "directional backward glue pouring" solution. Directional water-filling (Training Optimization for Energy Harvesting Communication Systems; Optimization of Energy Harvesting MISO Communication System with Feedback) shows that under causality the optimal "water level" can only step upward over time. Finite batteries add an overflow ceiling (Iterative Dynamic Water-filling for Fading Multiple-Access Channels with Energy Harvesting). All of these papers value an action by a scalar channel gain known per slot. Our change is to replace the channel gain with the marginal reduction in 4D reconstruction error. That quantity is unknown in advance, depends on the tissue's motion, and is not additive across frames, since two frames from the same quiescent interval are largely redundant.

Research Program

The device-side problem is posed per time slot $t$, one video frame. Let $w_t \in \{0,1\}$ indicate a wake-up. Let $H_t$ be the energy harvested, $E_0$ the initial store, and $B_{\max}$ the battery capacity. Each awake slot costs a fixed overhead $\epsilon$ plus a capture-and-transmit cost $c$. The constraints are

$$\sum_{i \le t} (\epsilon + c)\, w_i \;\le\; E_0 + \sum_{i \le t} H_i, \qquad E_0 + \sum_{i \le t} H_i - \sum_{i < t} (\epsilon + c)\, w_i \;\le\; B_{\max} + \text{(overflow lost)}, \quad \forall t.$$

The objective is to choose $\{w_t\}$ to minimize $\mathcal{L}(\{w_t\})$. This is the photometric and geometric error of a 4D model trained only on the selected frames and evaluated on every frame. Following the processing-cost analysis, we sweep the ratio $\epsilon / c$ rather than asserting device power figures. The file contains no measured capsule power budgets, so all energy quantities are normalized.

The work has three parts. The first is an offline oracle that knows future energy arrivals and the true per-frame value. It turns the set function $\mathcal{L}$ into a tractable surrogate: frame value is the measured reduction in held-out error when a frame is added to a reference reconstruction, with diminishing returns inside a burst. The oracle then solves the causal problem by dynamic programming over battery levels. It tells us what the best achievable error is at each energy budget. It also tells us whether the optimal schedules really are bursty and threshold-shaped. If they are not, the core claim is wrong and we learn that early.

The second part is a causal policy. Its value estimate uses only past information: the receiver's current reconstruction residual on the last received frames, fed back over the downlink, and the on-board inertial signal. V2-SfMLearner shows that capsule inertial data carries motion information, fusing it with vision for ego-motion. We reuse that signal source as a cheap predictor of deformation rather than as a depth cue. The policy itself is the glue-pouring structure made online. It fires a burst of $k$ frames when predicted value times the burst gain exceeds a threshold that depends on the battery level. The threshold has two parameters, a minimum reserve and a slope, that are fit on training sequences. We also derive the reserve rule for correlated harvesting analytically on a stylized periodic model: deformation events follow a quasi-periodic contraction cycle, and harvest is a lagged copy of the motion magnitude.

The third part is the receiver. It runs an unmodified deformable reconstruction method on the irregular frame set. We use Endo-4DGS, which deforms 3D Gaussians with small networks and uses monocular depth priors, and Efficient Deformable Tissue Reconstruction via Orthogonal Neural Plane (Forplane), whose space-time planes interpolate smoothly between sparse timestamps. Both are given true timestamps. Keeping the receiver fixed means any gain comes from frame selection, not from a new reconstructor. Point tracking is evaluated with LiteTracker, whose strictly causal ring-buffer design suits streaming but which assumes dense consecutive frames. Its accuracy under sparse bursts is itself an open measurement.

Baselines

The do-nothing reference is the unconstrained pipeline that receives every frame. On EndoNeRF, the strongest reported photometric number in the file is HFGS at 38.14 dB PSNR (HFGS), with Forplane at 37.31 dB and EndoGS at 36.84 dB as reported there. These are ceilings, not competitors. We re-run Endo-4DGS and Forplane on all frames under our own evaluation protocol so that the energy-constrained runs share a reference. On the energy-constrained version of this task there is no reported result found on Lacuna, so every constrained baseline is run by us.

The naive energy baseline is uniform duty-cycling. It wakes every $n$-th frame, with $n$ set so that expected consumption equals expected harvest over the sequence. It represents the "downsample the frame rate" objection. A second baseline is greedy energy-neutral capture, which wakes whenever the battery can pay for a frame. This baseline spends energy as it arrives and never overflows. It is the natural engineering default.

The strongest competitor is motion-triggered capture. It wakes, singly or in fixed bursts, when the inertial magnitude crosses a threshold, subject to battery availability. It uses the same cheap signal our policy uses, so any win we show must come from the energy-causal threshold structure and reserve, not from the sensor. We also run a Lyapunov drift-plus-penalty controller, the standard online tool for harvesting networks (Stochastic Routing and Scheduling Policies for Energy Harvesting Communication Networks; Stochastic backpressure in energy harvesting networks). It is given the same value estimate and the battery level as its queue.

All policies see identical video, identical energy traces, and the identical receiver with identical training iterations. Every policy's free parameters (thresholds, burst length, Lyapunov weight) receive the same tuning budget on a held-out split of sequences. Beating the competitor means two things at matched total energy: lower held-out reconstruction error on frames inside the top quartile of ground-truth motion, averaged over traces, and no worse error on the remaining frames. The bar for "beating" is a gap exceeding the seed-to-seed spread of the receiver.

Two ablations isolate the key ingredients. Forcing single-frame wake-ups (burst length one) tests whether contiguity matters. Replacing the learned value with a constant, so that only battery dynamics drive decisions, tests whether value awareness matters. The offline oracle bounds how much room any causal policy has.

Why not just…?

Why not just duty-cycle at the average harvest rate? Static schedules ignore both the fixed per-wake-up cost and the battery ceiling. They over-draw in lulls and overflow in surges, as shown for communication by Throughput Maximization for an Energy Harvesting Communication System with Processing Cost and Iterative Dynamic Water-filling for Fading Multiple-Access Channels with Energy Harvesting. For vision, isolated spaced frames also lose temporal coherence. Colonoscopy 3D video dataset with paired depth from 2D-3D registration reports that multi-frame information improved registration translation accuracy by 55.6% and rotation by 60.4% over single frames. That gap is the cost of throwing away neighbors. The uniform baseline and the single-frame ablation measure this directly.

Why not just trigger capture on inertial motion? This is the real threat. If inertial magnitude predicts deformation well and harvest is uncorrelated with motion, a tuned motion trigger may come close to our policy. We will report it if it does. Our expectation of a gap rests on two mechanisms that a memoryless trigger lacks. The first is reserving energy ahead of correlated events. The second is choosing burst length and threshold from the battery state so that overhead is not wasted on low-value wake-ups. The correlated-versus-uncorrelated trace comparison settles this. If the gap appears only under correlated harvesting, the contribution narrows to the reserve rule, and we will state the scope that way.

Why not just use a standard Lyapunov online controller? Drift-plus-penalty methods are designed for long-run averages. They are known to trade optimality for queue size, and they treat each slot's reward as additive. Burst value is neither. We give the Lyapunov controller the same value signal and tune it equally. If it matches our policy, the conclusion is that a generic controller plus a good value estimate suffices, which is a useful result. The oracle tells us which of the two is closer to optimal.

Why not just compress harder and keep every frame? Compression lowers the per-frame transmit cost $c$ but not the fixed overhead $\epsilon$ of powering camera, illumination, and radio. The processing-cost result says that overhead alone forces bursts once it is large relative to harvest. Our $\epsilon / c$ sweep shows the regime where compression alone would suffice: small $\epsilon / c$, where all policies should converge.

Proposed Experiments

The video sources are EndoNeRF and StereoMIS for strongly deforming surgical tissue under tools, the multimodal capsule dataset from V2-SfMLearner for paired video and inertial signals, and SuPer for point tracking under deformation. C3VD is used only to calibrate pose sensitivity, since its phantoms do not deform. Energy traces come in three families. The first is uncorrelated inductive-style arrivals with random interruptions. The second is motion-correlated arrivals generated as a lagged, noisy function of inertial or optical-flow magnitude, standing in for kinetic harvesting from peristalsis. The third is adversarial anti-correlated arrivals. We sweep battery size and $\epsilon / c$.

Reconstruction metrics are PSNR, SSIM, and depth error where ground truth exists, computed on all frames and separately on high-motion frames. Tracking is scored with occlusion accuracy and position error on SuPer annotations. LiteTracker's reported 97.45% occlusion accuracy with dense frames serves as the dense reference.

Three checks are decisive. First, does the oracle's optimal schedule actually come out bursty and threshold-shaped as $\epsilon / c$ grows? If not, the glue-pouring analogy fails. Second, at matched energy, does the causal policy beat motion-triggered capture on high-motion frames, and does the margin grow under motion-correlated harvest as the reserve argument predicts? Third, how much of the oracle-to-causal gap is closed? This tells us whether value prediction, rather than scheduling, is the bottleneck worth attacking next. A negative answer on the second check, with a positive one on the first, would still be a publishable finding: burst structure is necessary, but a simple trigger captures it.