Diagnostic Outage: Predicting When Rater Disagreement Flips a Radiographic Threshold

Many orthopedic decisions come down to whether one number measured on an X-ray crosses a cutoff. Examples include a spinal curve angle above 10°, a sacral tilt of at least 3°, or a shoulder gap of 6 mm or less. Experts measuring the same image routinely disagree by several degrees. Current automated systems report one number per image and summarize human disagreement as a single average error. They never say which images are likely to land on the other side of the cutoff for a different reader. This proposal builds a per-image model of how a new expert's measurement would be distributed, and from it the probability that the clinical decision flips. Borrowing the wireless term, we call this the image's "diagnostic outage probability."

Wireless engineers solved a structurally similar problem long ago. A satellite or laser link fails when the received signal drops below a threshold. Its error is modeled in layers: a fixed aiming offset, random wobble around it with different strength in different directions, a slowly fluctuating strength of the direct path, and random switching between line-of-sight and blocked conditions. Landmark-based angle measurement has the same three ingredients. Each expert has a consistent personal offset. Each placement wobbles more along a bone edge than across it. And readers sometimes disagree about which vertebra defines the curve, which produces a discrete jump rather than small noise. We propose to import this layered error structure, not the special-function mathematics, into Cobb angle measurement. We will test whether it predicts held-out experts' threshold decisions better than simpler uncertainty methods when only a handful of experts are available.

Core Claim

The disagreement between raters on a derived radiographic angle is not an image-independent Gaussian. It is a mixture with predictable structure. The continuous part grows with the across-edge uncertainty of corner landmarks and shrinks with the length of the bone edge they define. A discrete part switches on when two candidate end vertebrae have nearly equal tilt. A model that encodes this structure, with a handful of interpretable parameters, should estimate per-image threshold-flip probabilities more accurately than constant measurement-error bands or black-box variance regression, especially when trained on few raters. If a generic heteroscedastic network matches it at equal rater counts, the claim fails.

The Error Model

The model has three layers. The first layer is a fixed aiming offset per rater. In maritime laser links, Jung et al. split beam pointing error into a fixed misalignment (called boresight) plus jitter around it, with possibly unequal variance across axes. Depending on the case, the resulting error follows a Rician, Hoyt or one-sided distribution. We reuse this decomposition at the landmark level, without the closed-form performance analysis. Each corner landmark that rater $r$ places is displaced by a rater-specific mean offset $b_r$ (their systematic style) plus jitter.

The second layer is anisotropic jitter: random wobble expressed separately along and across the endplate. An endplate is the flat top or bottom surface of a vertebra. Because two corners define each endplate line, the tilt error of that line is approximately

$$\delta\theta \approx \frac{e_{\perp,2}-e_{\perp,1}}{L},$$

where $e_{\perp,i}$ is the displacement of corner $i$ perpendicular to the endplate and $L$ is the corner-to-corner length. Along-edge jitter therefore barely matters, while small upper-thoracic vertebrae amplify across-edge jitter. This is a falsifiable, image-specific prediction that an isotropic noise model cannot make.

Following the Shadowed-Rician idea in Yue et al., the direct-path strength there is itself random rather than constant. Here, the rater offsets $b_r$ are treated as draws from a population distribution. This lets the model predict a new rater, not only the raters seen in training.

The third layer is a regime switch over which vertebra is chosen to end the curve. Probabilistic line-of-sight models (Mozaffari et al., Zhang et al.) give the chance of an unobstructed path as an S-shaped curve of geometry. Azari et al. show that fading parameters must change with elevation rather than stay fixed. By analogy, the probability $\pi(x)$ that a rater picks a different end vertebra is a sigmoid in the tilt margin $\Delta\tau(x)$ between the top two candidates in image $x$. This gives

$$p(\theta_{\text{new}}\mid x)=(1-\pi(x))\,p_{\text{cont}}(\theta\mid x)+\pi(x)\,p_{\text{alt}}(\theta\mid x),\qquad \pi(x)=\sigma\!\left(\frac{a-\Delta\tau(x)}{s}\right).$$

The symbols are defined as follows. $p_{\text{cont}}$ is the distribution of the angle under the agreed vertebra choice, obtained by propagating layers one and two through the angle computation. $p_{\text{alt}}$ is the same quantity computed with the alternative vertebra. $\sigma$ is the logistic function, and $a$ and $s$ are a learned offset and scale. The diagnostic outage for cutoff $t$ is then $P_{\text{out}}(x)=\Pr(\theta_{\text{new}}\ge t\mid x)$ under this mixture, or the reverse inequality for a "less than" cutoff.

The per-image inputs are the landmark locations, local endplate orientation, $L$ and $\Delta\tau$. A landmark network supplies these, together with predicted across- and along-edge jitter scales. The global parameters are the offset-population spread and the sigmoid's $a$ and $s$. Everything is fit by maximum likelihood on multi-rater annotations. Monte Carlo sampling computes the probabilities. The closed forms from the fading literature are used only as a fast check and to interpret each layer.

Relation to Prior Work

Two papers motivate the gap. Deep learning automates Cobb angle measurement compared with multi-expert observers compares a segmentation-plus-polynomial pipeline against seven expert readers. It reports a mean deviation of 4.17° against a human intra-reader discrepancy of 5.16° and an ICC above 0.96. It treats expert spread as one aggregate error to beat, which is exactly the summary we replace with per-image distributions.

Seg4Reg+ and B-spine supply strong point estimators. Seg4Reg+ couples segmentation with angle regression, and B-spine fits a spline centerline to make estimates robust on low-quality images. Both are trained and scored against single reference annotations. We use one of them as the source of the point estimate and landmarks; we do not compete on point accuracy. The knee alignment work in Deep Learning-Based Alignment Measurement in Knee Radiographs names having only a single expert rater as its main limitation. That confirms multi-rater landmark data is the missing ingredient across this literature.

Pionono supplies the closest method. It models each rater as a Gaussian in a latent space, with the mean as bias and the variance as inconsistency, and it is evaluated on histopathology segmentation overlap. We adopt its bias-versus-inconsistency split but move it from pixel masks to landmark geometry. We add the anisotropic and regime-switch layers it lacks, and we score it on threshold decisions rather than overlap.

PENGWIN 2024 is a pelvic fracture benchmark. It finds 2D X-ray segmentation far behind CT (0.774 versus 0.930 IoU-F) because overlapping structures blur boundaries. This constrains the proposal: we model uncertainty in where boundaries sit, not attenuation physics.

Threshold-driven papers motivate the clinical framing. Examples include acromiohumeral distance (the shoulder gap) of 6 mm or less in Egli et al. and sacral slant of at least 3° in Joo et al.. They also mark where the method should transfer next. Nothing in the research file connects fading models to either medical direction.

Research Program

The work has three parts.

The first part is data. No dataset in the file has multi-rater, rater-identified, repeated landmark annotations on radiographs. We will therefore have six spine readers annotate the 128 test images of the AASCE 2019 set (the MICCAI scoliosis challenge, which provides 68 vertebral corner landmarks and three Cobb angles per image) in two sessions several weeks apart. The same protocol will be applied to a subset of the low-quality JLU-CJUH radiographs used by B-spine, to test behavior under degradation. Readers record both the landmarks and their chosen end vertebrae, so the regime-switch layer is directly observable.

The second part is the model. We will train a landmark network on the AASCE training split. Its point estimate will be Seg4Reg+ or B-spine, whichever reproduces better. We will add heads for across- and along-edge jitter scales. We will then fit the population and switch parameters on the multi-rater data with leave-one-rater-out cross-validation.

The third part is analysis. We will check each layer's prediction directly. Across-edge jitter should dominate along-edge jitter. Angle spread should scale like $1/L$. End-vertebra switching should follow a sigmoid in tilt margin.

Baselines

The naive reference is a constant measurement-error band. Every image gets a Gaussian around its point estimate, with a single standard deviation estimated from training raters, which is the standard-error-of-measurement view implied by ICC reporting. It is the do-nothing option clinicians already use implicitly. It gets the same point estimator and the same raters. Beating it means a lower held-out-rater negative log-likelihood and a lower Brier score on threshold flips.

The empirical alternative is a per-image bootstrap over training raters. For images in the training set, it uses the training raters' actual spread. For new images, it uses a nearest-neighbor transfer of that spread. It makes no structural assumptions, so it is the fairest test of whether structure helps at all.

The strongest learned competitor is a heteroscedastic regressor trained with the same backbone, data and tuning budget. It predicts a per-image angle mean and variance (or a two-component mixture) directly from the image using all rater annotations. A deep-ensemble version of Seg4Reg+ is run alongside it.

We also include a Pionono-style latent-rater landmark model. It uses an isotropic latent offset and Monte Carlo propagation to the angle, and it isolates what anisotropy and switching add over existing rater modeling.

Three ablations remove anisotropy, the regime switch and the population layer one at a time.

For point accuracy, the file reports a SMAPE of 7.32% for Seg4Reg+ on AASCE, MAE of 4.17/4.59/4.20 for B-spine on JLU-CJUH, and 4.17° mean deviation from experts in the multi-expert Cobb study. We require our point estimate to stay within reproduction noise of these.

On the actual question, no reported result was found on Lacuna for calibrated per-image threshold-flip probability on any radiographic benchmark. We must therefore beat our own runs of the baselines above. Beating them means a statistically significant gain in held-out-rater log-likelihood and flip Brier score, with no loss in point accuracy.

Why not just…?

Why not just Monte Carlo the raw landmark noise instead of borrowing fading models? We do use Monte Carlo. The fading literature contributes the layered structure (fixed offset, anisotropic jitter, random offset population, regime switching), not the Meijer-G closed forms. Given identical layer parameters, sampling and the analytic route give the same answer. The claim at stake is whether the structure, however computed, generalizes better than unstructured noise. The isotropic-latent baseline and the anisotropy ablation test exactly that.

Why not just train a network to predict per-image variance directly? This is the real threat. With many raters and images, a flexible heteroscedastic model may learn the $1/L$ effect and the end-vertebra ambiguity on its own. Our bet is that with six raters and roughly a hundred multi-rated images, the explicit $L$ and $\Delta\tau$ dependence is far more data-efficient. We will settle this by training both models on 2, 3, 4 and 6 raters. If the direct regressor matches us at six raters, we will report the structured model as a low-data method, not a general improvement.

Why not just report ICC or a standard error band, as clinical papers do? The multi-expert Cobb study shows the limitation itself: an ICC above 0.96 coexists with a mean intra-reader discrepancy of 5.16°. A single band cannot tell a large lumbar curve from a borderline thoracic one with small vertebrae and two equally tilted end candidates. The naive baseline quantifies how much image-specific prediction adds. If that gain is negligible, the project's premise is wrong.

Why not just use Pionono or STAPLE-style consensus? Consensus training throws away the disagreement we need. Pionono keeps it but models pixel masks with isotropic latent offsets and has no mechanism for discrete end-vertebra disagreement. Since end-vertebra disagreement plausibly drives the largest angle jumps, the landmark-adapted Pionono baseline measures that gap directly.

Proposed Experiments and Decisive Checks

The primary evaluation is leave-one-rater-out on the multi-rated AASCE test images. For each held-out rater, we score the log-likelihood of their two-session angles. We also score the calibration and Brier score of $P_{\text{out}}$ at the 10° cutoff used in the multi-expert study's F1 evaluation, and at bracing and surgical cutoffs fixed in advance with clinical collaborators. Finally, we measure AUROC for detecting images where the held-out rater's decision differs from the panel majority. The same evaluation is repeated on the JLU-CJUH subset to test whether predicted jitter grows appropriately on hazy images.

Three checks decide the project. First, on the observed data, across-edge landmark spread must exceed along-edge spread, and angle spread must correlate with $1/L$; if not, the anisotropy layer is unjustified. Second, observed end-vertebra disagreement must rise sigmoidally as tilt margin shrinks, and removing the switch layer must measurably worsen flip calibration on near-threshold images. Third, the full model must beat the heteroscedastic regressor at low rater counts. If all three hold, transfer is tested on a small multi-rater shoulder set at the 6 mm acromiohumeral cutoff. There, the one-sided, single-axis offset-plus-jitter case from the laser-link model applies directly. If the third check fails, the honest conclusion is that structure is useful for interpretation but not for prediction, and we will report it that way.