Spawning on Surprise: Budgeted Visual Sub-Agents for Long Surgical Video

In minimally invasive surgery the camera replaces the open wound. Procedures such as laparoscopic or transvaginal hysterectomy (vNOTES hysterectomy) or flexible ureteroscopy (RIRS for lower pole stones) give the surgeon, and any assisting software, nothing to work with except a video stream that runs for an hour or more. Asking a vision-language model (a language model that also reads images, which are cut into hundreds of "visual tokens" per frame) a question about such a procedure is expensive. The token count grows into the millions, and attention cost grows with the square of sequence length. The usual fixes compress the video once, for one model, before any question is asked. Most of a surgery is uneventful, though, and the few seconds that matter, such as an out-of-order step, a bleed, or a tool swap, are short and spatially small.

We propose a different allocation. A cheap, always-on monitor watches heavily compressed video and flags moments where the procedure departs from its expected course. Only at those moments does a coordinating language model start a short-lived helper agent, a "sub-agent." The sub-agent receives a dense, question-specific slice of visual tokens from the flagged window, answers a narrow sub-question, returns a short text verdict plus a few evidence tokens, and is discarded. The claim to be tested is specific. Under a fixed total budget of visual tokens per question, spending those tokens through several narrow sub-agent contexts that are created only when the monitor fires beats spending the same tokens in one context. It also beats a fixed, hand-built agent pipeline.

Core Claim

The three source directions play different roles here. Long minimally invasive surgical video is the application regime. Visual token explosion is the binding constraint, expressed as a hard per-question budget $B$, the total number of visual tokens that enter any language-model call while answering a question. The mechanism comes from work on dynamic multi-agent communication. In that work the set of agents and who talks to whom are decided at runtime instead of being fixed when the system is designed.

Two prior systems define the gap. AOrchestra has a central orchestrator create sub-agents on the fly as a tuple of instruction, curated context, tools and model. It shows that a curated context beats passing on the full history, but it works only on text, code and web tasks. SurgRAW brings multiple agents to surgery and beats a supervised Surgical-VQA model by 14.61% accuracy on SurgCoTBench. It does this with a fixed two-stream pipeline over single frames, with no temporal context and no token management.

Our proposal differs from both. The curated context is visual: a spatiotemporal selection of tokens with a budget. The decision to spawn is driven by a surgical-process signal rather than by the orchestrator's text reasoning alone. The test asks whether splitting the context into separate sub-agents matters, beyond simply choosing which tokens to keep.

That last point is the one a skeptic should press. If a single model shown exactly the union of tokens our sub-agents saw does just as well, then the multi-agent structure contributes nothing. In that case the contribution shrinks to "surprise-triggered token allocation." We treat that outcome as a live possibility and build the experiment around it.

Research Program

The system has three parts, all built on one open video vision-language backbone so that every comparison shares weights.

The monitor. This is a surgical phase recognizer run on video compressed with STIM-TM. STIM-TM is a training-free token-merging method that merges static background aggressively and protects moving instruments. It cuts compute by over 65% while keeping 86.04% accuracy on AutoLaparo. We reuse STIM-TM as the monitor's input stage rather than competing with it.

The monitor emits a surprise score at each step, taken from its own predictions:

$$s_t = -\log p(\hat{y}_t \mid \hat{y}_{t-1}) + \lambda\, H(p_t)$$

The symbols are defined as follows:

  • $p_t$ is the monitor's phase distribution at step $t$.
  • $\hat{y}_t$ is its predicted phase.
  • $p(\hat{y}_t \mid \hat{y}_{t-1})$ is a phase-transition prior estimated from training procedures.
  • $H$ is entropy.
  • $\lambda$ trades off "unexpected transition" against "uncertain frame."

This addresses a known weakness. Video Active Perception selects "surprising" frames by comparing them against a general text-to-video generator, CogVideoX, and that generator has no surgical tissue physics. Our surprise signal instead comes from procedural statistics. Those statistics match the failure mode LoViT reports: temporal models degrade when phases occur out of sequence.

The orchestrator. This is a language model that sees the question, the monitor's compressed phase timeline, and the list of flagged windows. For each sub-question it decides whether to answer from the timeline or to spawn a sub-agent.

The sub-agents. Each spawn specifies three things: a narrow instruction, a time window, and a visual token allotment $b_i$, with $\sum_i b_i \le B$. Within its window, the sub-agent's tokens are chosen by text-conditioned saliency in the style of AdaV. AdaV keeps 94.35% of LLaVA-NEXT-7B performance while dropping 95% of visual tokens, but it has only been tested on natural images. We combine AdaV-style saliency with STIM-TM's static-region score so that small instrument tips are not discarded.

Sub-agents return a verdict, a confidence, and an explicit "could not resolve" flag. Requiring that flag targets the silent-failure problem noted in THREAD. Two hard limits, a maximum spawn depth and a maximum spawn count, guard against the over-delegation loops described in ReDel.

We first run the orchestrator with in-context examples only. A later, optional stage fine-tunes it on logged traces, following AOrchestra's recipe.

The evaluation set. No benchmark in our sources asks questions about whole procedures. SurgCoTBench is frame-level, and Surg-QA (LLaVA-Surg) uses short clips. We therefore build a long-procedure question set from existing annotations on AutoLaparo and Cholec80. AutoLaparo has 21 hysterectomies with phase, camera-motion and instrument-mask labels; Cholec80 has phase and tool labels. The questions are generated from templates over these labels. They ask when a phase occurred, which instrument was present during a given phase, whether phases occurred out of order, and what the camera did before a given step.

We add controlled anomalies by splicing segments out of order within real procedures. Answers to the spliced versions are known by construction, which gives a direct test of whether surprise-triggered spawning finds rare events.

Baselines

Naive reference. Uniform temporal downsampling to a frame rate chosen so that the total visual tokens equal $B$, with all frames passed to one call of the same backbone. This is the standard practice in surgical datasets that sample at 1 fps. It shows what any allocation scheme must beat for free.

Strongest single-model competitor. One model sees query-guided, STIM-TM-protected tokens chosen across the whole video at the same total budget $B$. That is, AdaV-style selection plus STIM-TM's static-region protection in a single context. It is the strongest competitor because it uses everything our system uses except multiple agents and the monitor.

The closest reported numbers are these:

  • STIM-TM: 86.04% accuracy on AutoLaparo, but for phase recognition, not question answering.
  • LoViT: 81.43% video-level phase accuracy on AutoLaparo and 91.50% on Cholec80.

There is no reported result on Lacuna for long-procedure question answering on AutoLaparo or Cholec80. We will report our own numbers for this baseline.

Fixed-topology baseline. A SurgRAW-style pipeline: SurgRAW's fixed instrument and action specialists, run over uniformly spaced windows at the same budget. Its strongest reported number is a 14.61% accuracy gain over Surgical-VQA on SurgCoTBench. We rerun the released code with our backbone rather than GPT-4o, so the backbone is held constant.

Key ablation. This one decides the central claim. We log the exact token sets our sub-agents received and feed their union, with timestamps, to one model in one call. A second ablation keeps sub-agents but spawns them at random windows instead of surprise-flagged ones. Together these separate the value of the topology from the value of the trigger.

Fairness conditions. All arms share:

  • the backbone,
  • the budget $B$, counting every visual token in every call,
  • prompt-tuning effort, matched by giving each arm the same number of development iterations on a held-out split of procedures.

Language-model text tokens and wall-clock latency are reported separately. Spawning is not free in either, so neither is hidden.

What counts as winning. We beat the strongest single-model competitor only if both conditions hold:

  • higher accuracy at matched $B$ at two or more budget levels, on the anomaly questions and the full set, with confidence intervals computed across procedures;
  • no worse accuracy on ordinary questions.

Why not just…?

Why not just give one strong model good query-guided token selection? This is the real threat, and the union ablation exists to settle it. The case for separate contexts is indirect. AOrchestra found that curated context beats full context in agents, and THREAD found that isolating sub-task context prevents clutter. Both results are on text. If the single-context union matches our system, we will report that the gain comes from surprise-triggered allocation and drop the multi-agent claim.

Why not just use a fixed agent pipeline like SurgRAW, plus compression? A fixed pipeline spends the same effort on every window. Its design also cannot express "look harder here because the procedure just went off-script," which is exactly the case LoViT identifies as hard. The fixed-topology baseline at equal budget, scored separately on spliced out-of-order procedures, tests this directly. If it matches us on the anomaly questions, dynamic spawning is unnecessary.

Why not just feed a long-context frontier model the whole video? SurgRAW reports large gains from structured decomposition over plain GPT-4o on surgical frames. MovieChat documents memory of about 200MB per frame for uncompressed video models. Even so, a frontier model at unlimited budget is a useful upper reference on a subset of procedures. We will run it there and report how close budgeted systems get, rather than assume it fails.

Why not just use a supervised phase model for phase questions? For pure phase recognition it will win. LoViT's 81.43% on AutoLaparo is the bar, and we do not claim to exceed it. Our system uses such a model as its monitor. The question set deliberately asks compositional questions, such as which tool appeared during an out-of-order phase, that a phase classifier cannot answer alone.

Proposed Experiments

Main experiment. We sweep budget $B$ over several levels on the AutoLaparo and Cholec80 long-procedure questions and plot accuracy against budget for every arm. Separate curves are reported for:

  • ordinary versus spliced-anomaly questions;
  • peak GPU memory, latency and language-model text tokens per question.

Decisive check. Compare our system with the single-context union ablation at matched $B$. A positive gap on anomaly questions that holds across budgets supports the topology claim. A null gap means the contribution reduces to the trigger.

Second check. Compare surprise-triggered spawning with random-window spawning. This shows whether the procedural surprise signal finds the events that matter. We also measure monitor recall on spliced anomalies directly.

Parity checks. Without retuning, we run the sub-agent component on two existing benchmarks:

  • SurgCoTBench, against SurgRAW's reported 14.61% gain;
  • Surg-QA, against LLaVA-Surg's reported GPT score of 2.45 and Accuracy@1 of 0.545.

These confirm that budgeted sub-agents do not lose short-context ability.

Exploratory test. On AutoLaparo camera-motion prediction, the reported baseline is about 29.8% with ResNet-50 plus LSTM. The original authors attribute the low score to unobserved intent. We will test whether phase-conditioned sub-agents add anything there and will report the result whether or not they help.