Systems that watch surgical video and write the operative note are graded today with metrics built for machine translation. These metrics count shared words, so a generated note that says "the specimen bag was intact on transanal retrieval" scores almost perfectly against a reference that says the bag tore. We propose an automatic score for operative reports that behaves the way a surgeon reading the note would. It breaks each report into an ordered list of surgical events of the form instrument, action, target, outcome. It compares that list with the reference and charges each mismatch according to how dangerous it would be to believe it. It also penalizes events that are reported in an impossible order, such as a duct cut before it was clipped. We judge the score by one test: whether it agrees with surgeons' counts of clinically meaningful errors.
We test the score hardest on natural orifice specimen extraction (NOSE), a colorectal and gynecological technique. The organ is removed laparoscopically and then pulled out through the anus or vagina instead of through a new abdominal cut. NOSE suits a stress test for three reasons. Its benefits are real: a pooled analysis of 1,787 patients found fewer overall complications and roughly 87% fewer wound complications than abdominal-incision extraction (He et al. meta-analysis). Its risks sit in a few short events whose order and outcome matter enormously: bagging the specimen, sealing the bag, passing it through the stump, closing the stump, and deciding to convert to an incision. None of these events exists in any current evaluation vocabulary. A metric that handles NOSE has to model order, polarity and consequence. A metric that only handles cholecystectomy frames might not.
Core Claim
Operative-report quality is determined mainly by three things. The first is whether the few safety-critical events are present. The second is whether their outcome or polarity is stated correctly. The third is whether they appear in a feasible order. A score built on those three components will track surgeon error counts better than lexical and embedding metrics. It will also beat off-the-shelf radiology metrics. We state openly that a rubric-prompted large language model judge is a serious competitor, and the experiments are designed to settle that comparison rather than assume it away.
The new element is not "a clinical metric for surgery" in the abstract. It is the combination of three parts:
- a temporally ordered event representation taken from surgical vision work;
- consequence tiers adapted from hallucination-severity work;
- weights fitted to surgeon error counts, following the RadCliQ recipe.
The resulting metric is validated on a new surgeon-annotated error benchmark that includes a NOSE slice.
Where This Sits
Three lines of prior work motivate the gap.
- Report generation graded by word overlap. Wang et al. generate reports from robotic nephrectomy video using a scene graph of tool–tissue interactions. They report BLEU-1 0.6994 and CIDEr 3.9006, a 36.6% CIDEr gain over the prior method, but never measure clinical correctness.
- Assistants graded by an unconstrained model judge. LLaVA-Surg builds a surgical video assistant and grades free-text answers with a GPT score. The authors themselves note that the assistant still produces confident hallucinations.
- Vision work that never becomes text. SSG-Com, dynamic scene graphs and CholecTriplet2021 show that surgical understanding needs action triplets, hand identity and temporal context. They stop at classification, graded by mAP and accuracy.
The CholecTriplet challenge also supplies a key motivating asymmetry. Instruments are recognized at about 80% average precision, but anatomical targets only at about 46%. Generated notes will therefore most often get wrong what tissue was cut, which is exactly the error a word-count metric weights least.
Three papers from the clinical-metric literature supply the method.
- Parsing plus a fitted composite. RadGraph F1 and RadCliQ parse reports into entity–relation graphs and fit a composite to radiologist error counts, reaching Kendall τ_b 0.522. That paper also found that strong generators beat random report retrieval by only 5.5% once measured properly. We reuse both the fitting recipe and the retrieval sanity check.
- Category-aware partial credit. RaTEScore adds synonym-aware entity matching and a learned penalty for confusing abnormal with normal findings. It reaches τ 0.527 on ReXVal and 67% accuracy on synonym-versus-antonym tests, against 14% for BERTScore. We reuse its affinity-with-penalty idea.
- Severity tiers. Med-HallMark ranks hallucinations into five tiers from catastrophic to minor. We adapt these tiers to operative consequences.
These radiology metrics also constrain the design. They describe static findings, have no instruments or actions, and, as RaTEScore's authors note, do not model relations between entities. Our metric therefore adds these where they lack them, rather than retraining their vocabularies.
Research Program
The event schema. We define an operative event schema that extends the CholecT50 vocabulary of instrument, verb and target with two fields:
- an outcome/state field, such as intact or torn, patent or occluded, closed or open;
- a phase tag.
We also add a NOSE extension covering specimen bag, protective sleeve, rectal or vaginal stump, colpotomy, circular stapler anvil, purse-string suture, extraction route and conversion to auxiliary incision. Surgeons assign every event type a consequence tier (catastrophic, critical, attribute, minor), adapted from Med-HallMark. They also specify a small set of precedence constraints, for example "clip before transect" and "bag sealed before transluminal passage."
The parser. A parser turns each report into an ordered event sequence. We build it two ways, so that parsing quality is separated from scoring:
- a fine-tuned token-classification model in the style of RaTEScore's entity recognizer;
- a prompted language model constrained to the schema.
The score. Scoring compares the reference sequence $R$ with the candidate sequence $C$ after matching events by affinity:
$$S = \frac{\sum_{(e,\hat e)\in M} w(e)\,a(e,\hat e)}{\tfrac12\left(\sum_{e\in R} w(e)+\sum_{\hat e\in C} w(\hat e)\right)}\,\bigl(1-\lambda D\bigr)$$
The symbols are defined as follows:
- $M$ is the set of matched event pairs.
- $w(\cdot)$ is the weight of an event's consequence tier.
- $a(e,\hat e)\in[-1,1]$ is an affinity that gives partial credit for synonyms and near-miss targets. It goes negative when the outcome polarity is reversed, so that "bag torn" versus "bag intact" is worse than omission.
- $D$ is the fraction of matched event pairs under a precedence constraint that appear in the wrong order.
- $\lambda$ sets the order penalty.
Unmatched candidate events act as hallucinations, and unmatched reference events act as omissions. Both are charged through the denominator at their tier weight. The tier weights and $\lambda$ are fitted on a training split of surgeon error counts, as RadCliQ does, and are frozen before test.
The benchmark. The validation benchmark has two parts.
The general part covers three sources, with candidate notes drawn from several generators:
- EndoVis 2018 nephrectomy captions (Wang et al.);
- reference notes assembled from CholecT50 triplets and phases;
- Endoscapes-SG201 scene graphs, which add hand identity.
The generators are:
- a Wang et al.-style scene-graph captioner;
- LLaVA-Surg;
- a general video-language model;
- nearest-neighbor report retrieval.
The NOSE part draws on two sources. The first is colorectal and gynecological procedures in the WebSurg-derived Surg-QA corpus, screened for transanal or transvaginal extraction. The second, if that screen yields too few cases, is a small partner-hospital set of NOSE-LAR and transvaginal hysterectomy recordings, the latter motivated by vNOTES. For each video a surgeon writes a reference note focused on extraction. At least two surgeons then annotate every candidate note with error counts by tier, following the ReXVal protocol. We report agreement between the surgeons.
Baselines
Naive reference points. Two references anchor every comparison. The first is BLEU/ROUGE/CIDEr as used by Wang et al., which is the current practice in surgical report generation. The second is BERTScore. Both are computed on identical candidate–reference pairs. No reported result was found on Lacuna for their correlation with surgeon judgment on operative reports; the radiology analogue is BLEU Pearson 0.27 on RaTE-Eval. Retrieval sits alongside as a "do nothing" generator: if a metric cannot separate real generators from retrieval, it is not measuring clinical content.
Existing clinical metrics, unmodified. We run RaTEScore and RadCliQ as released, plus AlignScore, which scores whether each candidate claim is supported by the reference (paper). These are the strongest published domain-aware metrics. Their reported numbers are τ 0.527 for RaTEScore on ReXVal and τ_b 0.522 for RadCliQ. No reported result was found on Lacuna for either metric on surgical text.
The strongest competitor: a rubric-prompted language-model judge. This judge receives the reference, the candidate, our full event schema, the tier definitions and the precedence constraints, and is asked for tiered error counts. It gets the same development split and the same number of prompt iterations that we spend fitting weights. It is scored on the identical held-out test set. The number it must be read against is LLaVA-Surg's reported ρ = 0.94 between its GPT score and one surgeon. That figure was measured on question-answering items rather than operative notes, and with a single rater.
Ablations. Three ablations isolate the key ingredients. Unweighted event F1 (tiers set equal, $\lambda=0$, polarity treated as mismatch rather than negative) is a RadGraph-style score transplanted to surgery. The other two remove severity weighting alone and removing the order term alone. Every ablation uses the same parser as the full metric.
What counts as winning. Beating a baseline means a higher Kendall τ with surgeon error counts on the held-out test set, with a bootstrap confidence interval that excludes the baseline's value. It also means passing the perturbation tests described below.
Why not just…?
Why not just prompt a strong language model to grade the note? This is the real threat. LLaVA-Surg's ρ = 0.94 shows a model judge can track a surgeon at coarse granularity. Against that, Med-HallMark reports that general judges are inconsistent across repeated runs and tend to go along with false premises, whereas its supervised detector reaches 93.88% multi-round consistency. The fair comparison gives the judge our schema, so the only difference left is whether scoring is structured and deterministic. The experiment settles the question, with the decision rule fixed in advance. If the rubric judge matches our τ within the confidence interval and passes the polarity and order perturbations with repeat-run stability, we will recommend it. In that case we would report the schema, the tiers and the benchmark as the contribution, and the deterministic metric would become an auditing tool. We will not argue for extra machinery the data do not support.
Why not just run RaTEScore or RadGraph on surgical notes? Their vocabularies have no instruments, actions, bags or stumps. RaTEScore does not model relations between entities, so "hook dissected cystic duct" and "grasper retracted cystic duct" become the same bag of entities. The unmodified-metric baseline measures how much this costs. The order-swap and target-swap perturbations should expose it directly.
Why not just grade the upstream vision model with triplet mAP? Frame-level mAP never sees the note. It cannot tell whether a correctly detected event was omitted from the report, stated with the wrong outcome, or placed in the wrong order. The report-level analysis tests this directly. We will measure how well a generator's triplet mAP predicts its surgeon error count. A weak relation would show that the two measure different things.
Proposed Experiments
Perturbation suite. Starting from reference notes, we apply controlled edits: flip outcome polarity (bag intact versus torn, stump closed versus open), swap targets (cystic duct versus common bile duct), reverse a precedence pair, delete a conversion event, and paraphrase with synonyms. A good metric must score the paraphrase above every harmful edit and must rank catastrophic edits below minor ones. We report pairwise accuracy in the style of RaTEScore's synonym/antonym test for every metric and judge.
Agreement with surgeons. On the general split and the NOSE split separately, we compute Kendall τ against surgeon error counts. Weights are fitted only on non-NOSE training data. This checks whether the tier structure transfers to extraction events or whether NOSE requires its own calibration.
Re-ranking existing generators. We score all generators, including retrieval, under CIDEr and under the new metric. Following RadCliQ's finding in radiology, we ask whether apparent CIDEr gains shrink toward the retrieval baseline once safety errors are counted.
Deciding the result. The decisive outcome combines the NOSE-split τ and the perturbation pass rate. If both beat unweighted event F1 and the rubric judge, weighting by consequence and order is the ingredient that matters. If the judge ties, the benchmark stands and the metric becomes an auditing tool.
References
- Meta-analysis of laparoscopic anterior resection with natural orifice specimen extraction (NOSE-LAR) versus abdominal incision specimen extraction (AISE-LAR) for sigmoid or rectal tumors
- Dynamic Interactive Relation Capturing via Scene Graph Learning for Robotic Surgical Report Generation
- LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Lecture Learning
- Towards Holistic Surgical Scene Graph
- Dynamic Scene Graph Representation for Surgical Video
- CholecTriplet2021: A benchmark challenge for surgical action triplet recognition
- Evaluating progress in automatic chest X-ray radiology report generation
- RaTEScore: A Metric for Radiology Report Generation
- Detecting and Evaluating Medical Hallucinations in Large Vision Language Models
- Transvaginal natural orifice endoscopic surgery for hysterectomy: a prospective cohort study
- AlignScore: Evaluating Factual Consistency with A Unified Alignment Function