Clock-Anchored Moving Horizon Estimation for Surgical Event Anticipation

Systems that forecast surgical workflow try to tell an operating team how long it will be until the next phase begins or the next instrument is needed. They get worse the further ahead they look, and their frame-by-frame forecasts disagree with each other in a way that is physically absurd. If a model says at 10:00 that clipping starts in 3 minutes, then at 10:01 it should say 2 minutes, not 4. The true quantity being forecast obeys an exact law: until the event happens, remaining time falls by one second per second. Current anticipation networks on the Cholec80 laparoscopic cholecystectomy benchmark ignore this law. Each frame's output is an independent guess, so the noise in those guesses is never pooled.

We propose to enforce that law at inference time using moving horizon estimation (MHE). MHE is a state estimator from control engineering that, at every time step, re-solves a small constrained optimization over a sliding window of recent measurements, and it summarizes everything older in a compact "arrival cost." Two changes do most of the work. First, we re-express each forecast target as an absolute event time, which stays constant between plan changes. Second, we treat the network's per-frame forecasts as noisy, censored measurements of that event time. MHE then becomes a tiny convex program per target. It pools evidence across frames, survives smoke and occlusion, and admits hard ordering constraints, while costing negligible compute next to the video network. The question is whether this flattens the curve of error against forecast distance, which is the prediction horizon decay that limits current surgical anticipators.

Core Claim

Surgical anticipation suffers from horizon decay because per-frame regressors throw away a free physical invariant, not only because they lack capacity. Towards Graph Representation Learning Based Surgical Workflow Anticipation builds tool-interaction graphs fed to a causal temporal convolution network and reports anticipation error that grows from 2- to 5-minute horizons. SUPR-GAN generates future phase sequences autoregressively and finds accuracy degrading beyond a 15-second sweet spot, along with "temporal logic errors." CoStoDet-DDPM co-trains a diffusion branch to regularize features but discards that branch at inference, so the deployed model has no online corrective mechanism. All three address decay through training-time architecture or losses. We instead add an inference-time estimator that reconciles a network's outputs with the dynamics the target must obey.

Our claim is specific. On Cholec80 anticipation, an MHE layer has four parts: event times anchored to the clock, a one-sided (censored) treatment of forecasts saturated at the horizon cap, sparse rather than Gaussian revisions, and an arrival cost. Applied to the same frozen network predictions, it should reduce in-horizon mean absolute error, with the largest gains at mid-to-long remaining times. It should beat both a Kalman filter and a plain moving average of event-time estimates, and it should degrade more gracefully under simulated visual dropouts.

The Estimator

Let $y_k^{(j)}$ be the network's forecast at frame $k$ (at 1 fps) of the remaining time until event $j$, clipped at the horizon $h$ as in the standard Cholec80 anticipation protocol. Let $T_k^{(j)}$ be the latent absolute time of the next occurrence of event $j$, so the implied remaining time is $T_k^{(j)} - k$. Between plan changes $T$ is constant. When an instrument is used and then needed again, or when the surgeon's plan shifts, $T$ jumps. At frame $t$, with window length $N$, we solve:

$$ \min_{T_{t-N:t}} \; \mathcal{Z}(T_{t-N}) + \sum_{k=t-N}^{t} w_k\, \ell_h\!\left(y_k,\, T_k - k\right) + \lambda \sum_{k=t-N}^{t-1} \left|T_{k+1} - T_k\right| \quad \text{s.t. } T_k \ge k,\; T^{(a)}_k \le T^{(b)}_k \text{ for ordered pairs } (a,b). $$

Here $\ell_h$ is a Huber penalty on $y_k - (T_k - k)$ when $y_k < h - \epsilon$. When the forecast is saturated, $\ell_h$ becomes the one-sided penalty $\max(0,\, h - (T_k - k))$, because a saturated forecast only says "at least $h$ away." The weights $w_k$ are per-frame measurement confidences, $\lambda$ sets how reluctant the estimator is to accept a plan change, and $\mathcal{Z}$ is a quadratic arrival cost centered on the previous window's estimate of $T_{t-N}$. The L1 term is the deliberate departure from Gaussian process noise. It makes the estimated event time piecewise constant, so the estimator averages aggressively when nothing changes and jumps cleanly when something does.

Every term is convex, so each step is a small linear or quadratic program over roughly $N \times J$ scalars, with $J$ the number of targets. This sits squarely in the setting where Constrained linear state estimation - a moving horizon approach established MHE as a quadratic program with an arrival cost and proved stability under observability. We reuse that formulation. The ordering constraints are imposed only for event pairs whose order is never violated in the Cholec80 training split. This is the kind of hard inequality that MHE handles natively and recursive filters do not, as Rao et al. and the thermal-constraint work of Kleyman et al. show in physical settings.

The weights $w_k$ come in two versions. One is fixed and tuned on validation. The other is predicted per frame from the network's features by a small head trained through the estimator. That second version follows Neural Moving Horizon Estimation for Robust Flight Control, which learns MHE weighting matrices online by differentiating through the estimator and cut force-estimation error by up to 76.7% relative to NeuroBEM.

The expected form of the gain explains why this targets horizon decay rather than generic noise. If per-frame errors were independent with spread $\sigma$, then after $n$ frames without a plan change the error in the event time would shrink like $\sigma/\sqrt{n}$, as for any average of $n$ independent measurements of a constant. Targets far in the future have been inside the window for more frames by the time they matter, so they should gain the most. Real errors are strongly autocorrelated, because adjacent frames share visual features. The realized gain will therefore be smaller, and measuring the effective number of independent frames is itself one of our analyses.

Research Program

The work has three stages. First, we reproduce two anticipation backbones on Cholec80 and freeze their per-frame outputs: the graph-plus-temporal-convolution model of Zhang et al., and a ConvNeXt-LSTM deterministic branch of the kind used in CoStoDet-DDPM. Second, we implement the estimator above as a causal post-processor and evaluate it at horizons of 2, 3 and 5 minutes for both phase and instrument targets. Third, we make the confidence weights learnable end-to-end and test robustness when frames are removed or corrupted.

This mirrors the outage experiment in Optimization-Based System Identification and Moving Horizon Estimation Using Low-Cost Sensors for a Miniature Car-Like Robot. There, MHE kept a continuous trajectory through a 1.127-second positioning blackout (about 68 missed updates), even though its nominal accuracy matched a well-tuned extended Kalman filter. We take that last finding seriously: it predicts that our advantage over a well-tuned filter may appear mainly under degradation, not on clean frames.

The arrival cost replaces the hand-built "history memory" of Aggregating Long-Term Context for Learning Laparoscopic and Robot-Assisted Surgical Workflows. That model concatenates fixed HMM and cumulative-likelihood statistics to an LSTM and raised short-phase accuracy on MGH100 from 12.5% to 31.2%. Our version is an optimized summary of past evidence about each event time, and it keeps per-step cost constant regardless of surgery length. That addresses the latency growth that LoViT reports for full-history attention.

The broader motivation, that data-driven forecasters decay at long lead times when they lack physical invariants, comes from WEATHER-5K and from LT3P, whose physics conditioning brought 72-hour typhoon displacement error to 143.03 km. Our invariant is far humbler, the wall clock, but it is exact.

Baselines

The naive reference points are the raw per-frame network outputs, which are what every published Cholec80 anticipator reports, and a constant predictor that always outputs the training-set mean remaining time clipped at $h$. Any gain must first show up as a drop from the raw outputs. The constant predictor tells us how much of the remaining error is irreducible.

The strongest existing competitor is the graph-based anticipator of Towards Graph Representation Learning Based Surgical Workflow Anticipation. It reports an instrument in-horizon mean absolute error (inMAE) of 1.27 at a 5-minute horizon (versus 1.48 for IIA-Net) and a phase inMAE of 0.54 at 2 minutes (versus 0.62). For CoStoDet-DDPM the file gives only a relative 16% reduction in remaining-time error and no absolute value, so we treat it as a second backbone rather than a numeric target. Because our method post-processes predictions, the fair comparison is the same frozen predictions, on the same splits (40 training videos, held-out test videos), with and without the estimator. Beating the baseline means a lower inMAE than the reproduced raw backbone at every horizon, with a paired per-video test. We also report whether the reproduced backbone matches the published 1.27 and 0.54 so that readers can judge the absolute level.

The simple smoothers share the same predictions and receive the same validation tuning budget, a grid over their few hyperparameters. They are a causal moving average of $k + y_k$ over the same window $N$, which converts forecasts to event times and averages them; a scalar Kalman filter on event time with constant dynamics, with saturated outputs either dropped or clipped; and a causal median filter on raw outputs.

The ablations isolate each ingredient of the estimator. We swap the L1 revision term for a squared one, replace the censored loss with an ordinary Huber loss on clipped outputs, remove the arrival cost, remove the ordering constraints, sweep $N$ from 10 to 300 frames, and compare learned weights against fixed ones. The claim survives only if the full estimator beats the moving average and the Kalman filter by a margin larger than the across-fold spread.

Why not just…?

Why not just average $k + y_k$ over a window? This is the real threat, and it may capture most of the gain, because the core insight is the change of variable to event time. Averaging fails in two specific places. When the target is beyond the horizon, the saturated outputs carry only a lower bound, and averaging them biases event times early. After a plan change, a window average lags by roughly $N/2$ frames, whereas the L1 term allows a clean jump. We will report error separately on frames near saturation and within 60 seconds of a ground-truth revision. If the moving average matches MHE on both, we will report that the clock reparameterization alone is the contribution and drop the optimization machinery.

Why not just use a Kalman filter? A Kalman filter with constant event-time dynamics is the unconstrained, Gaussian special case of our estimator, and Rao et al. show the two coincide in that regime. It cannot represent a measurement that says "at least $h$," cannot enforce ordering between events, and with Gaussian process noise it must either smear over plan changes or trust noise. The Chronos results warn that on clean data a filter may tie. The censoring ablation and the dropout experiment are therefore where it should lose, and if it does not lose there, we will say so.

Why not just train the network with a temporal-consistency loss? A soft loss rewarding $y_{k+1} \approx y_k - 1$ during training is cheap. However, it constrains only what the network learned on training videos, and it offers nothing at test time when smoke or occlusion corrupts inputs. That is the gap left by CoStoDet-DDPM discarding its corrective branch at inference. We will train this variant of the ConvNeXt-LSTM backbone under the same compute budget and compare it with and without the estimator. If the loss-trained network gains nothing further from MHE, the estimator is redundant.

Why not just smooth phases with an HMM? HMM smoothing constrains discrete phase labels, not continuous time-to-event. The HMM baselines reported in SUPR-GAN reach only 31.5% per-transition accuracy on Cholec80 against SUPR-GAN's 60.9%. A duration-aware HMM could in principle produce remaining-time estimates, but it would replace the strong visual regressor rather than correct it. We therefore include it only as a reference point and do not regard it as the main competitor.

Proposed Experiments

The primary experiment reports inMAE and out-of-horizon error for phases and instruments at 2, 3 and 5 minutes on Cholec80. This covers both backbones, the full estimator, every simple smoother and every ablation. The key figure is a horizon-decay curve: absolute error binned by true remaining time. The decisive check is whether the full estimator lowers the slope of that curve and not just its intercept, relative to both the raw outputs and the event-time moving average.

The second experiment tests robustness. We remove contiguous blocks of 10 to 120 frames from test videos, mimicking camera removal, and separately replace frames with blurred or darkened versions, mimicking smoke and lens fouling. We then measure the excess error during and after each corruption. Here MHE should keep coherent forecasts by running the clock forward, while the raw network and the averages degrade.

The third experiment measures the autocorrelation of per-frame errors to estimate the effective number of independent frames. This tells us whether the observed gain is near the ceiling implied by pooling.

The fourth experiment repeats the main protocol on AutoLaparo, whose 66-minute average videos stress the arrival cost. We found no reported anticipation result on Lacuna for AutoLaparo, so comparisons there are internal only.

We also report the per-frame solve time and compare it with the 0.030-second-per-frame inference of the graph model, to confirm that the estimator adds negligible latency.

The study counts as a success if three things hold: the full estimator reduces 5-minute instrument and phase inMAE against the raw backbone on both backbones; it beats the event-time moving average specifically on saturated and post-revision frames; and it halves excess error under dropouts relative to the Kalman filter. If the moving average matches it everywhere, the honest conclusion is that a one-line reparameterization explains most of Cholec80's apparent horizon decay, and that finding alone would be worth publishing.