Scoring Surgical Skill as a Function of Procedure Time

Automated surgical skill assessment tries to reproduce what a faculty surgeon does when filling in a rating form such as OSATS (Objective Structured Assessment of Technical Skills): watch a whole procedure and give a few scores. Current video models do not watch whole procedures. They cut the video into short clips, typically 16 to 64 frames, and run a 3D convolutional network on each clip. A 3D convolutional network is an image network whose filters also slide along time, so it sees motion as well as appearance. The models then merge the clip outputs by voting, averaging, or a recurrent network. This is the same mistake medical imaging made with CT scans. Processing each slice independently and stitching the results together throws away the continuity between slices, and restoring it with a simple smoothing step across slices raised pancreas segmentation Dice from 56% to 68% (Deep convolutional networks for pancreas segmentation in CT imaging). A procedure is a "volume" whose third axis is time, and snippet-wise skill models discard its through-time context.

We propose to treat each recorded performance as one continuous curve over normalized procedure time, built from per-clip 3D convolutional features and tool trajectories. We then predict the rating with a functional regression: the score is an integral of that curve against a learned, smooth weight function $\beta(u)$. The weight function is shared across all surgeons and says which parts of the procedure drive the score. The bet is concrete. Surgical skill datasets are tiny, with JIGSAWS having eight surgeons and the clinical gastrectomy sets 20 to 57 videos. In that regime, a smoothness prior over procedure time should generalize to unseen surgeons better than recurrent or attention-based aggregation. As a side effect, it gives a time-resolved account of each score that educators can inspect and that we can test.

Core Claim

Classical functional data analysis supplies the model: scalar-on-function regression, where a scalar outcome is related to a whole curve through a weight function. For performance $n$, let $z_n(u)$ be a low-dimensional feature curve over normalized time $u \in [0,1]$ and $T_n$ its duration. We fit

$$ y_n = \alpha + \gamma \log T_n + \int_0^1 \beta(u)^\top z_n(u)\, du + \varepsilon_n , $$

where $y_n$ is the rubric score, $\alpha$ and $\gamma$ are scalars, and $\beta(u)$ is the weight function. We penalize roughness with $\lambda \int_0^1 \lVert \beta''(u) \rVert^2 du$, where $\lambda$ sets how smooth $\beta$ must be.

Two design choices carry the idea. First, time is normalized so that procedures of different lengths line up. Duration is kept as an explicit covariate because completion time is itself a strong skill signal, and normalizing it away would silently delete that signal. Second, each rubric item $j$ gets its own weight function $\beta_j(u)$. A question like "where does the rhexis-formation score come from?" then has a falsifiable answer.

The claim has three parts. Under leave-one-user-out evaluation, this model should match or beat discrete aggregation built on the same 3D convolutional backbone. Its advantage should grow as training data shrinks. Its learned $\beta_j$ should place weight on procedure segments that matter for item $j$ more often than chance.

Research Program

Building the feature curve. We run a Kinetics-pretrained I3D or R(2+1)D backbone over overlapping clips and record each clip's feature vector at its center time. Kinetics pretraining is non-negotiable: without it, I3D accuracy on JIGSAWS fell to 41% in Video-based surgical skill assessment using 3D convolutional neural networks. We reduce the channels with functional principal component analysis, which finds the few dominant modes of variation across curves. We then project each mode onto a penalized basis over $u$.

We will compare B-splines with piecewise Chebyshev polynomials, borrowing the latter from CHESS, which reports that Chebyshev bases avoid edge oscillations and resample across sensor rates better than linear interpolation. CHESS also found that forcing continuity across segment boundaries destabilized training, so we start with one global basis per curve and use piecewise variants only if needed. Where tool trajectories exist, we fit them in the same basis and append them to $z_n(u)$. JIGSAWS provides robot kinematics; for video-only data we track keypoints as in Video-based assessment of intraoperative surgical skill.

Fitting the score. Because the curve and $\beta$ share a basis, the integral reduces to a matrix product with the basis inner-product matrix. Fitting the linear model is therefore penalized ridge regression, which is cheap and has a closed form. In a second stage we allow nonlinearity using the functional-network layers of Nonlinear Functional Modeling Using Neural Networks. Those layers lowered error over functional linear models on its benchmarks: RMSE 0.822 versus 2.517 on fat-spectrum data. We will fine-tune the backbone end to end only if the frozen version shows a gain, since the small datasets make full fine-tuning risky.

The depth question. Volumetric context loss enters a second way. Laparoscopic and robotic video is a 2D projection of a 3D field, and the file contains no measurement of how much skill information depth carries. JIGSAWS logs true 3D tool positions, so we can measure this directly. We fit the functional model on full 3D trajectories, and again on trajectories with the axis nearest the camera's viewing direction removed. That axis is estimated from the kinematics, since the file gives no calibration. The drop in Spearman correlation between the two settings is an estimate of how much skill signal monocular video loses. A small drop is a useful negative result: it would say that depth estimation is not where effort should go.

Baselines

The naive reference points are two. The first is a ridge regression on log duration alone. The second is a ridge regression on duration plus time-averaged backbone features. If the functional model cannot beat the duration-only score, its other signals are not adding much.

The hand-crafted kinematics baseline follows OR Vision. It computes on the order of 150 kinematic summaries per trial and feeds them to a Random Forest. That paper reports precision of 0.82–0.84 across datasets, and accuracy and F1 of 0.92 on UTSW, which we cannot access. It is a strong, explainable competitor. We run it on identical trajectories and identical splits.

The strongest existing method is ViSA (Surgical Skill Assessment via Video Semantic Aggregation). ViSA groups R(2+1)D features into semantic clusters and runs a bidirectional LSTM per group. It reports Spearman 0.90 on JIGSAWS, against 0.79 for C3D-LSTM, and MAE 1.27 on HeiChole, against 1.33–1.42 for CNN-LSTM baselines. The file does not say which cross-validation scheme produced the 0.90. We will rerun ViSA under both leave-one-supertrial-out and leave-one-user-out, using the same backbone and clip sampling as our model, and tuning its cluster count $K$ over the same budget we spend tuning $\lambda$ and the basis size.

Attention baseline. We also train a small attention-pooling transformer over the same clip features, parameter-matched to our nonlinear variant. It is the most direct test of whether learned per-clip weighting already does what $\beta(u)$ does.

Other reported numbers. For the clinical regime, the Unified Multi-Path model (Towards Unified Surgical Skill Assessment) reports Spearman 0.80 on JIGSAWS and 0.565 on 20 gastrectomies. On capsulorhexis, KP-TCN reports AUC 0.788 overall and 0.72 for rhexis formation. For leave-one-user-out on JIGSAWS, no reported result was found on Lacuna; Funke et al. state only that performance decreased.

What counts as winning. Beating ViSA means higher Spearman under leave-one-user-out on JIGSAWS with the same backbone, with a paired bootstrap over held-out surgeons excluding zero difference. A tie under leave-one-supertrial-out is acceptable, since that split mostly rewards memorizing individual surgeons.

Ablations. These isolate the key ingredient. We set the roughness penalty to zero. We replace the integral with an unconstrained MLP on the basis coefficients. We swap normalized time for raw clock time. We drop the duration covariate.

Why not just…?

Why not just put a transformer or ViSA's LSTM over the clip features? This is the real threat. Attention also yields per-time weights, and ViSA already beats simple pooling. Our answer is not that attention cannot represent the function. It is that, with eight JIGSAWS surgeons, a population-level smooth $\beta(u)$ has far fewer effective degrees of freedom and should overfit individual surgeons' habits less, which is the failure Funke et al. observed under leave-one-user-out. The parameter-matched transformer and the training-size sweep settle this. If the transformer matches us at every data size, the functional structure buys only interpretability, and we will report it as such.

Why not just compute kinematic summaries and use a Random Forest? OR Vision shows this is competitive and explainable. However, summaries such as total path length cannot say when a deficit occurs. Kinematics alone also miss field and tissue context, which multi-path clinical models found necessary (Towards Unified Surgical Skill Assessment). Running our model on kinematics only versus kinematics plus video features measures the second point directly.

Why not just interpolate onto a fixed grid and use a temporal convolutional network? Interpolation handles dropped frames and mismatched rates, and KP-TCN is a strong model. But a temporal convolutional network still pools over a fixed receptive field and has no population-level time weighting. CHESS reports that linear resampling loses up to 12.5% to Chebyshev resampling, though on sensor data rather than surgery. We include an interpolate-then-temporal-convolution baseline and a frame-dropping stress test, in which 10–50% of frames are removed at random, to see which approach degrades more gracefully.

Proposed Experiments

Primary benchmark. JIGSAWS is the primary benchmark: Suturing, Needle Passing, and Knot Tying, scored by Spearman correlation with global rating scores, under both validation schemes. HeiChole, scored by MAE, tests transfer to real cholecystectomy. Capsulorhexis and the gastrectomy sets will be added if access is obtained. Those let us check per-item $\beta_j$ against rubric sub-scores, and compare model agreement with the human junior-surgeon range of 0.41–0.62 reported in Surgical Skill Assessment on In-Vivo Clinical Data via the Clearness of Operating Field.

Decisive check: small data. We train on progressively fewer surgeons under leave-one-user-out. The prediction is that the gap over ViSA and the transformer widens as surgeons are removed. If it does not, the core claim fails.

Decisive check: attribution. This check tests whether $\beta$ is faithful. For each held-out video, we mask the procedure segments where $|\beta|$ is largest and, separately, segments of equal length where it is smallest. Masking the high-weight segments should move the predicted score significantly more. On JIGSAWS we also report how $\beta$ mass is distributed across the dataset's gesture annotations, as a descriptive check that the weights land on interpretable actions rather than idle frames.

Secondary study: depth. The 3D-versus-projected-trajectory comparison quantifies how much skill signal is lost when the surgical field is reduced to a 2D image. It is reported as a measurement, whichever way it comes out.