Deep learning models that find nerves in ultrasound images are meant to help anesthetists place local anesthetic beside a nerve. Papers on these models usually report accuracy over thousands of image frames. Those frames come from a much smaller number of people, and frames from the same person, often from the same probe sweep, are close to copies of one another. A single needle biopsy can misrepresent a heterogeneous tumor because it samples one small region. In the same way, a benchmark built from many frames of a few necks or thighs samples a narrow slice of human anatomy. If the benchmark is scored as though every frame were an independent patient, the effective sample size is overstated, confidence intervals are too narrow, and differences of one Dice point between architectures can look real when they are not. We propose to measure this effect on the public ultrasound-guided regional anesthesia (UGRA) benchmarks, to turn it into concrete rules for how many subjects and how many frames per subject a study needs, and to use those rules to settle two contradictions in the published literature.
In two sentences: treat each patient in an ultrasound dataset the way a pathologist treats one biopsy core, as a correlated cluster of observations rather than many independent ones, and then count how much real evidence the benchmarks contain. With that count, we can say which published improvements survive, how many new patients a study must recruit to detect a given gain, and whether annotation money should go to more frames per patient or to more patients.
Core Claim
Our claim is that, on current UGRA benchmarks, the between-subject correlation of model errors is large enough that the effective number of independent observations is close to the number of subjects, not the number of frames. If so, several reported architectural gains sit inside the subject-level noise. One example is region-aware pooling raising Dice from 70.71% to 74.23% on the Kaggle Ultrasound Nerve Segmentation set in Region-aware Global Context Modeling for Automatic Nerve Segmentation from Ultrasound Images. Another is the finding in Investigation of Appropriate Scaling of Networks and Images for Convolutional Neural Network-Based Nerve Detection in Ultrasound-Guided Nerve Blocks that downsampled 384×384 inputs beat full resolution.
The standard survey-sampling quantity captures the idea. If a test set has $N$ frames spread over subjects with $m$ frames each on average, and $\rho$ is the intraclass correlation (the fraction of per-frame error variance that is shared within a subject), then the effective sample size is
$$ n_{\text{eff}} = \frac{N}{1 + (m-1)\rho}. $$
This is Kish's design effect for cluster sampling. When $\rho$ is moderate and $m$ is in the hundreds, $n_{\text{eff}}$ collapses toward the subject count. The same logic gives the cost-optimal number of annotated frames per subject, roughly $m^\ast = \sqrt{(c_s/c_f)(1-\rho)/\rho}$. Here $c_s$ is the cost of recruiting and scanning one more subject and $c_f$ is the cost of annotating one more frame. What is new is not the formula. It is measuring $\rho$ for UGRA models and testing whether the allocation rule, derived for estimating means, also holds for training deep networks. Nobody has checked this.
The biopsy literature motivates the framing. Coexpression of the IGF-IR, EGFR and HER-2 is common in colorectal cancer patients shows that one core misjudges receptor status because of within-tumor variance. Habitat-based radiomics analysis for evaluating immediate response in colorectal cancer lung metastases treated by radiofrequency ablation shows that summarizing a lesion by its whole-tumor mean hides informative sub-regions: whole-lesion radiomics gave a test AUC of 0.769, against 0.870 for the habitat signature. We borrow both lessons. Subjects are the unit of sampling, and a single subject should be summarized by the distribution of its frame scores, not only the mean. The method we reuse for that summary comes from Multiple Instance Learning for Heterogeneous Images: Training a CNN for Histopathology. That paper aggregates instance predictions by sorted quantiles from the 10th to the 90th percentile instead of max-pooling, which raised histologic subtype accuracy from 89.8% to 95.2%. We apply the same quantile aggregation to evaluation rather than to training. Clinically, the 10th-percentile Dice within a patient (the moments when the nerve is lost) matters more than the average.
The power literature constrains the analysis. HAI and NAI titer correlates of inactivated and live attenuated influenza vaccine efficacy reports an 85.5% efficacy trend that missed significance (p = 0.20) because few informative events occurred. Comparison between Onyx and coil embolization for persistent type 2 endoleaks after endovascular aneurysm repair reports sac expansion of 20% versus 51.6% with P = 0.472 in 46 patients. Both show large numerical gaps that small effective samples cannot resolve. Ultrasound work has the same problem in plain view. Label Uncertainty for Ultrasound Segmentation collected 718 videos but evaluated readmission on 4 patients and could not reach significance. Our position is that frame counts in UGRA papers hide the same weakness less visibly.
Research Program
The first component is a subject-level audit. For each benchmark where subject identity can be recovered, we train the reference models under subject-grouped five-fold cross-validation, so that no subject appears in both training and test folds. We then fit a mixed-effects model to per-frame Dice or detection F1, with a random intercept per subject and, for video data, a nested random effect per sweep. This gives $\rho$, the design effect, and $n_{\text{eff}}$. The file does not say how many subjects underlie the 11,143 Kaggle segmentation images or the 619-image Kaggle peripheral nerve set. Recovering subject identifiers from dataset metadata is therefore the first deliverable. If a dataset has no identifiers, we will cluster near-duplicate frames by image similarity as a proxy and report the audit as a lower bound on correlation. The 101-volunteer brachial plexus dataset from the scaling study is the cleanest case if access can be obtained, because its subject structure is explicit. The adductor canal and femoral clips from Temporal Monte Carlo Dropout for Robust Uncertainty Quantification: Application to Point-of-Care Ultrasound-guided Nerve Blocks give the video case, with strong within-clip correlation.
The second component is a comparison protocol. Two models are compared by a paired, subject-level cluster bootstrap on per-subject quantile profiles (10th, 50th and 90th percentile frame score). From the estimated variance components we produce power curves giving the number of test subjects needed to detect a difference of 1, 2 or 4 Dice points with 80% power. This turns "is 74.23 better than 73.34?" into a question with a stated answer and a stated recruitment requirement.
The third component tests the allocation rule for training, not just for evaluation. At a fixed annotation budget we draw training sets that trade subjects against frames per subject, from many subjects with few frames each to few subjects with many frames each. We then measure generalization to held-out subjects. If the curve peaks near the $m^\ast$ predicted from $\rho$, data collectors get a usable rule. If it does not, that is itself informative: it would mean redundancy within a subject helps learning, through augmentation-like effects, in a way it cannot help estimation.
Baselines
The naive reference point is the evaluation most papers implicitly use. It is a random frame-level train/test split with a frame-level bootstrap or no confidence interval at all. We run it on exactly the same models, checkpoints and data, so that every difference between protocols comes from the evaluation alone. Our audit beats this reference if frame-level intervals cover the subject-level truth substantially less often than their nominal 95%. We estimate that coverage by repeatedly holding out random subsets of subjects and checking whether the frame-level interval contains the held-out performance.
The strongest existing segmentation method is the region-aware pyramid aggregation and adaptive pyramid fusion network of Region-aware Global Context Modeling for Automatic Nerve Segmentation from Ultrasound Images. It reports a Dice coefficient of 74.23% on the Kaggle Ultrasound Nerve Segmentation set, against 70.71% for its ResNet-34 U-Net baseline and 73.34% with the region-aware pooling module alone. The strongest detection method is YOLOv4-P7 at 384×384 input from Investigation of Appropriate Scaling of Networks and Images for Convolutional Neural Network-Based Nerve Detection in Ultrasound-Guided Nerve Blocks. It reports F1 of 94.7% on the public Kaggle peripheral nerve set and 80.8% on the 101-volunteer set. We retrain both with their published settings and an identical tuning budget for every arm, with tuning done only inside training folds. "Beating" them is not the aim here. Their reported numbers are claims to be re-adjudicated. The audit succeeds if we can state, with subject-level intervals, whether each reported increment survives.
The fair methodological competitor is ordinary subject-grouped cross-validation with a subject-level bootstrap on mean scores. This is good practice already and the obvious thing a careful reviewer would do. Our additions are three: the measured variance components and power curves, the quantile profile in place of the mean, and the allocation experiment. We include an ablation that keeps grouped cross-validation and removes only the quantile profile. That isolates whether lower-tail per-subject reporting changes model rankings. The quantile profile earns its place if it reverses or separates rankings that mean Dice leaves tied.
Why not just…?
Why not just use subject-grouped splits and stop there? Grouped splits remove leakage, and we adopt them in every arm. They do not say whether a 0.9-point difference is detectable with the subjects available, how many subjects a new study must recruit, or whether to spend annotation effort on more frames or more people. The variance components answer those questions, and grouped splitting alone does not produce them. If the audit finds $\rho$ near zero, this alternative is enough and our contribution reduces to a reassuring measurement. We will report that outcome plainly if it occurs.
Why not just collect more annotated images? If $\rho$ is substantial, the design-effect formula says that extra frames from existing subjects add almost nothing to $n_{\text{eff}}$. The allocation experiment tests directly whether this also holds for training. This is the alternative most likely to be partly right. Within-subject frames may help training through implicit augmentation even when they add no evaluation evidence, and the experiment is designed to show exactly that split if it exists.
Why not just report frame-level bootstrap confidence intervals, which some papers already do? A frame-level bootstrap treats correlated frames as independent and should under-cover. The held-out-subject coverage test measures by how much. Published precedents of large numerical gaps that small effective samples could not resolve, as in the vaccine-correlate and endoleak studies above, suggest the error is not cosmetic.
Why not just fit a mixed-effects model and call it standard statistics? That is what we do, and the statistics are textbook. The open questions are empirical. Nobody in this literature has published $\rho$ for nerve-segmentation errors. Nobody has re-scored the resolution-versus-pooling contradiction at the subject level. Nobody has tested whether cluster-sampling allocation rules hold for training networks.
Proposed Experiments
The first decisive check targets the contradiction between full-resolution regional pooling and input downsampling. We train both families on each available dataset across three input sizes under subject-grouped folds. We then ask whether either advantage has a subject-level 95% interval that excludes zero. Two outcomes are possible, and both are informative. Either one approach wins robustly and the contradiction is resolved, or neither does and the contradiction is shown to be unresolvable at current sample sizes. In the second case we give the number of subjects needed to resolve it.
The second check re-scores the label-handling disagreement, which current subject counts may also be unable to settle. Label Uncertainty for Ultrasound Segmentation favors hard confidence thresholding. On the Within-class Variation Issue in Alzheimer's Disease Detection favors soft targets, reporting balanced accuracy of 0.8608 on ADReSS. We implement both on the Kaggle segmentation masks, using annotator disagreement where available and boundary distance as a proxy otherwise, and compare them under the subject-level protocol.
The third check is the fixed-budget allocation sweep described above. Success means the observed optimal frames-per-subject falls within a factor of two of $m^\ast$ computed from the audit's $\rho$ and plausible cost ratios.
The last step is to carry the variance components into trial planning for regional anesthesia itself. Trials such as The effect of ultrasound-guided intercostal nerve block on postoperative analgesia in thoracoscopic surgery: a randomized, double-blinded, clinical trial (119 analyzed patients, rescue analgesia 9.8% versus 31%) will increasingly use AI guidance as an intervention. We will report how many patients such a trial needs before a segmentation improvement of a given size could plausibly show up in clinical endpoints. No reported result was found on Lacuna for any subject-level effective sample size on a UGRA benchmark, which is the gap this work fills.
References
- Region-aware Global Context Modeling for Automatic Nerve Segmentation from Ultrasound Images
- Investigation of Appropriate Scaling of Networks and Images for Convolutional Neural Network-Based Nerve Detection in Ultrasound-Guided Nerve Blocks
- Coexpression of the IGF-IR, EGFR and HER-2 is common in colorectal cancer patients
- Habitat-based radiomics analysis for evaluating immediate response in colorectal cancer lung metastases treated by radiofrequency ablation
- Multiple Instance Learning for Heterogeneous Images: Training a CNN for Histopathology
- HAI and NAI titer correlates of inactivated and live attenuated influenza vaccine efficacy
- Comparison between Onyx and coil embolization for persistent type 2 endoleaks after endovascular aneurysm repair
- Label Uncertainty for Ultrasound Segmentation
- Temporal Monte Carlo Dropout for Robust Uncertainty Quantification: Application to Point-of-Care Ultrasound-guided Nerve Blocks
- On the Within-class Variation Issue in Alzheimer's Disease Detection
- The effect of ultrasound-guided intercostal nerve block on postoperative analgesia in thoracoscopic surgery: a randomized, double-blinded, clinical trial