Spine Segmentation Benchmarks Are Saturated on the Wrong Metric

Programs that outline each vertebra in a CT scan, called vertebral segmentation models, now score above 0.91 Dice on the main public benchmark. Dice is the voxel-overlap score between a predicted mask and the reference mask. Scores that high suggest the problem is solved. For opportunistic screening, the hope is that bone density can be read off CT scans taken for other reasons, and what matters there is different. The model has to put the right label on the right vertebra (L1 must really be L1), and the bone-density measurement inside that vertebra has to be right. A model can outline every vertebra almost perfectly and still shift every label by one level. That error leaves Dice nearly untouched and makes the screening result meaningless.

We propose to measure spine segmentation by its downstream consequence: how often a model's output would change the bone-density reading, and the screening decision, at the vertebra a clinician actually uses. We then ask whether that measure restores the ability to tell models apart on test splits deliberately built to be hard. As a secondary, strictly tested question, we ask whether a large language model acting as a judge can estimate these consequences on new hospital scans that have no expert labels. It must do so better than a simple anatomical rule checker, or we report that it does not.

Core Claim

Our claim is that the apparent saturation of vertebral segmentation leaderboards is mostly an artifact of the metric. Saturation here means top models cluster so tightly near the score ceiling that their differences fall below measurement noise. The claim has three testable parts.

First, on the same set of models, a statistical saturation index computed on Dice will be near its maximum. The same index computed on a level-coupled consequence metric will be well below it. The consequence metric is the absolute error in mean trabecular Hounsfield units (CT density) at the screening vertebra, plus the rate at which the screening decision flips.

Second, test splits built from how badly models fail when moving between anatomical phenotypes will widen the spread between models further. Examples of such phenotypes are transitional vertebrae (an extra or missing lumbar vertebra), compression fractures, metal hardware and thick-slice scans.

Third, these consequence-based differences are the ones that matter for screening. Dice differences between top models will sit below the disagreement between two human annotators. Consequence differences will not.

The literature motivates this claim from two sides. On the spine side, the VerSe benchmark describes the one-label shift: masks with good overlap carrying uniformly wrong level labels. CTSpine1K reports Dice close to zero on transitional L6 vertebrae and a large domain gap between its cohorts and VerSe. Neither paper asks whether its headline metric has stopped separating models.

On the measurement side, When AI Benchmarks Plateau defines a saturation index. The index compares the spread among the top five models to their standard errors and approaches 1.0 when the gaps fall below noise. That study covers only text benchmarks. We apply it to 3D dense prediction for the first time and use it to compare metrics, not benchmarks. Variations in Relevance Judgments and the Shelf Life of Test Collections argues that once models match human inter-annotator agreement, further gains fit annotator noise. That argument supplies our annotator-ceiling test.

The downstream metric is clinically meaningful because the steps after segmentation are known to be reliable. Ramschütz et al. show that automated vertebral density extraction tracks dedicated quantitative CT across scan parameters, so labeling and segmentation errors become the dominant remaining source of error. Population studies such as Pickhardt et al.'s CT biological-age model and Palm et al.'s 3117-vertebra height study consume these level-specific outputs directly.

Research Program

The first component is the consequence metric. For every scan, we take the vertebra the pipeline labels as the screening level and derive a trabecular region from it by a fixed, deterministic erosion of the vertebral body. We apply the identical procedure to the reference vertebra that truly carries that label. We use L1 as the primary level and fix it before any experiment; L2 serves as a robustness check. We record three quantities: the absolute difference in mean density between the two regions, whether a one-level or larger shift occurred, and whether the density crosses a pre-registered screening cutoff on one side but not the other.

Because the erosion and the density readout are identical for prediction and reference, this metric isolates what the segmentation model contributes. Errors from the density step itself, which Ramschütz et al. address, are held fixed.

The second component is a saturation audit. We train or obtain a pool of segmentation models spanning the families reported in VerSe and CTSpine1K: full-resolution 3D nnU-Net, cascaded 3D networks, patch-based 3D convolutional models and a 2D slice-wise model. Each is trained with several random seeds. For each metric we compute the saturation index from bootstrap standard errors over test scans, following the Akhtar et al. definition. We then compare three metrics: Dice, the VerSe identification rate, and our consequence metric.

We also have a subset of about 60 VerSe scans re-annotated by a second rater. This places every model gap relative to the inter-rater gap on the same metric.

The third component is hard splits. Following Patel et al., we build a cross-generalization error matrix: train on one group and record the error on each other group. Patel et al. used this procedure to cut malware-detection accuracy from 99.64% to about 72% on harder splits without new data. Our groups are anatomical and acquisition phenotypes: standard anatomy, lumbosacral transitional anatomy, moderate-to-severe compression fractures, metal hardware, slice thickness above a fixed threshold, contrast-enhanced scans, and the four source cohorts of CTSpine1K plus VerSe. The off-diagonal entries with the largest consequence error define a small number of hard splits, which we release with fixed seeds.

The fourth component is the label-free judge. It is the part of this proposal we expect could fail, and we say so plainly. MLLM-as-a-Judge finds two relevant failures in multimodal judges on 2D images: they often judge better from text descriptions than from pixels, and step-by-step visual reasoning can hallucinate premises. We therefore do not feed the judge raw volumes.

Instead, each candidate segmentation becomes a structured anatomical audit. The audit is a per-vertebra table of assigned label, centroid position along the spine, volume, anterior and posterior height, and adjacency to ribs and to the sacrum, all computed from the mask. It also includes mid-sagittal and coronal renders with the mask overlaid.

The judge sees two candidate segmentations of the same scan side by side. It must say which one mislabels the screening vertebra, or whether neither does. Each pair is run in both presentation orders, because position bias is documented in the long-form medical QA benchmark and in MLLM-as-a-Judge. Pairwise comparison is the format for which MLLM-as-a-Judge reports the highest agreement with humans.

Hard2Verify describes a "generosity bias": judges lose the ability to catch errors once a case exceeds their competence. It predicts that the judge will fail on transitional anatomy specifically, so we report the judge's error-catching rate separately on each hard split.

CT-Agent shows that a multimodal language-model agent can reason over 3D chest CT through token compression. It is a report generator, not an evaluator, and it never touches the spine. We borrow only its lesson that 3D input must be compressed before a language model can use it. Our compression is anatomical structure, not learned tokens.

Baselines

The naive reference point is reporting mean Dice on the standard VerSe and CTSpine1K splits and treating the leaderboard as settled. The strongest reported figures on VerSe'20 are a 91.72% Dice score and a 96.6% identification rate for the top performer (VerSe). For nnU-Net on CTSpine1K, no reported result was found on Lacuna for numeric Dice or 95th-percentile Hausdorff distance; we recompute these ourselves. For the saturation question, our metric "beats" Dice if, on the identical model pool and test scans, its saturation index is clearly lower, with bootstrap intervals that do not overlap.

The strongest existing competitor on the measurement question is the VerSe identification rate. It is level-aware, it already exists, and it might detect shifts just as well. We compute it on exactly the same predictions. The consequence metric earns its place only if it changes model rankings or separates models that identification rate ties.

For the label-free judge, the strongest competitor is a deterministic anatomical rule checker that we write ourselves and run on the same structured audit the judge sees. It checks whether the label sequence is contiguous, whether the count of rib-bearing vertebrae is consistent with the thoracic labels, and whether the last labeled vertebra borders the sacrum. It costs nothing and has no prompt to tune, which makes it the honest threat.

The second judge competitor is Quality Sentinel, a regression network that predicts Dice from image-mask pairs using CLIP text conditioning. It reports a correlation of r = 0.902 with true quality on abdominal organ datasets but was not validated on pathological spines. We retrain it on the same spine training data, with the same compute and the same number of hyperparameter configurations as we spend on prompt development. Prompt development is capped at a fixed number of variants and selected only on a validation split.

For spine-mask judging by any language model, no reported result was found on Lacuna. The closest figure, about 79% pairwise agreement with humans for GPT-4V, comes from 2D natural images (MLLM-as-a-Judge) and is context, not a target.

Two ablations isolate the key ingredient, which is the structured anatomical audit. The judge with renders only tests whether the text table is what helps. The judge with the table only tests whether the pixels add anything at all.

The judge beats its competitors only under one condition. On the transitional and fracture hard splits, it must catch screening-level mislabels at higher recall than the rule checker at the same false-alarm rate, with intervals that exclude a tie. Agreement in any easy-case condition does not count.

Why not just…?

Why not just use the identification rate VerSe already reports? This is a real threat. Identification rate is already level-aware, but it scores a vertebra as found when its centroid lies near the reference. It averages over all vertebrae, so a shift at the screening level counts the same as a missed upper cervical vertebra. It also says nothing about whether the density reading moved enough to change a decision. It is already at 96.6% on VerSe'20 and may itself be saturated. The saturation audit settles this directly. If identification rate restricted to the screening level yields the same rankings and saturation index as the consequence metric, we will say so and recommend that simpler metric. The hard splits and the annotator-ceiling analysis still stand in that case.

Why not just write anatomical rules instead of asking a language model? For clean scans with standard anatomy, we expect rules to be as good as the judge and much cheaper, which is why the rule checker is the primary judge baseline. Rules are underdetermined exactly where the VerSe and CTSpine1K papers report catastrophic failures. On transitional anatomy, counting from the skull and counting from the sacrum give different answers. Rib cues can conflict with lumbosacral morphology, so a fixed precedence order will be wrong on some fraction of cases. The per-split comparison will show whether the judge reconciles conflicting cues better or simply inherits Hard2Verify's generosity bias. If it does not reconcile them better, the judge component is a negative result, and we will report it as one.

Why not just retrain Quality Sentinel on spine data? It is efficient and well-correlated with Dice on abdominal organs, and with retraining it may estimate spine Dice well. But it is trained to predict Dice. A predictor of Dice inherits Dice's blindness to one-level shifts, because a shifted mask that is well outlined deserves a high predicted Dice. Our experiment asks it to rank pairs by screening-level mislabel rather than by overlap. We predict it will do well on Dice estimation and poorly on shift detection; if it does well on both, it is the preferable tool.

Why not just collect more scans or keep the test set private? Akhtar et al. find that private test sets do not prevent saturation. CTSpine1K grew the labeled pool from 374 to 1,005 scans and still reports a substantial gap with VerSe. More data of the same kind, scored with the same metric, would reproduce the same plateau. Hard splits reuse existing data to target the phenotypes where failures concentrate.

Why not just give a multimodal model the raw CT volume? The 2D evidence from MLLM-as-a-Judge already points toward text over pixels, and 3D volumes make the input-size problem worse, as CT-Agent's compression machinery shows. We include a render-only ablation so this is measured on spine data rather than assumed.

Proposed Experiments and Decisive Checks

All experiments use VerSe'19/'20 and CTSpine1K with their released splits, plus our hard splits. The first decisive check is the saturation comparison. Dice must show a high saturation index on the model pool while the consequence metric does not, on both the standard and hard splits. If both are high, models truly are near the ceiling for screening, and the claim fails.

The second check is the annotator ceiling. On the doubly annotated subset, Dice gaps between top models should fall within inter-rater Dice variation, while screening-decision flip rates between models should exceed inter-rater flip rates.

The third check is that the hard splits widen the spread between models on the consequence metric. The acceptance criterion, set in advance, is that the gap between the best and worst model in the pool grows by more than the bootstrap error.

The fourth check is the judge comparison described under Baselines. It is reported per split with order-swapped runs and with the error-catching rate shown separately from the rate of confirming correct outputs.

Secondary analyses report the consequence metric by patient sex and age group. The motivation is that Quality Sentinel found lower label quality for women and older patients on TotalSegmentator. Those are the populations in which osteoporosis screening matters most.