Blow-Up Profiles of Inter-Rater Disagreement

When several experts outline the same lesion or optic cup, their outlines never coincide, and current medical segmentation methods treat the resulting disagreement as a single per-pixel number: how many raters marked this pixel. That number mixes three kinds of disagreement. The first is hand jitter, which is clinically meaningless. The second is a consistent offset of the boundary, where one rater draws tighter than another. The third is disagreement about shape, where one rater draws a notch in the cup rim or a small nodule on a cyst wall and another does not. The third kind is often the diagnostically important one, and it is the kind that smoothing-based label fusion is most likely to erase.

We propose to separate these kinds using blow-up analysis, a technique from the study of partial differential equations. At a boundary point you zoom in progressively, rescaling coordinates by the zoom factor, and ask what shape survives in the limit. For a smooth curve the answer is a straight line, and in general the surviving shape is called the tangent cone. Raters who agree on the local tangent cone but not on position produce disagreement that decays predictably as the zoom window grows. Raters who disagree on the tangent cone itself produce disagreement that stays constant across scales. We add a thin formal layer from quantitative equational logic, a calculus of "these two things are within distance ε" statements. It supplies sound rules for how such scale-indexed disagreement bounds combine across raters and under weighted label fusion. We then use the decomposition to build fused training targets that remove jitter but keep structural disagreement.

Core Claim

Inter-rater disagreement at a boundary point has a measurable scaling signature. Let $A_i$ and $A_j$ be two raters' masks, $x_0$ a boundary point, and $B(x_0,\lambda)$ the disc of radius $\lambda$ around it. Define the scale-normalized disagreement

$$D_{ij}(x_0,\lambda)=\frac{|(A_i \triangle A_j)\cap B(x_0,\lambda)|}{|B(x_0,\lambda)|},$$

the fraction of the window on which the two raters disagree. This normalization is exactly the blow-up rescaling $y=(x-x_0)/\lambda$ applied to both masks. If the raters share a tangent cone and differ only by a boundary shift $\delta$, the disagreement region is a strip of area about $2\lambda\delta$ inside a disc of area $\pi\lambda^2$. Hence $D_{ij}\propto\delta/\lambda$, a scaling exponent of one, with saturation near $\lambda\approx\delta$. If they have different tangent cones, the disagreement fraction tends to a positive constant, giving an exponent of zero. Jitter and offset both scale with exponent one. They are told apart by sign coherence: the ratio of the net signed area difference to the unsigned area inside the window. This ratio is near one for a consistent offset and near zero for jitter.

The claim to be tested has two parts. First, the fitted exponent and the coherence ratio classify disagreement into jitter, offset and structure reliably on real multi-rater data. Second, a fusion target that suppresses only the jitter component preserves diagnostic signal better than frequency-based smoothing does.

Relation to Prior Work

Blow-up analysis supplies the method. Analytic regularity of a free boundary problem zooms in at interface points and shows that global constraints vanish, leaving a regular minimal-surface profile. Energy release rate for non-smooth cracks in planar elasticity shows that crack-tip rescaling washes out macroscopic roughness and leaves a universal square-root profile. Both results use blow-up to prove that irregularity disappears. We use the same rescaling for the opposite purpose: to detect which differences between curves do not disappear. Learning Heat-Based Equations in Self-Similar Variables shows that rescaled coordinates are a strong practical inductive bias for learning. That supports the bet that the right normalization matters, but that work concerns PDE extrapolation, not annotations.

The multi-rater literature motivates the problem. Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle (DiFF) fuses rater masks with spatially varying weights chosen to maximize downstream diagnosis. It found that the learned weights pick up high-frequency noise, and it adds a convolutional smoother to suppress high frequencies. Its own visualizations show that consensus fusion flattens asymmetric melanoma borders and underestimates the vertical cup-to-disc ratio. Those are exactly the structural features a frequency filter can also blur, because a small notch is high-frequency. Our jitter-only filter replaces the smoother with a scale-signature test.

Three other methods model the rater distribution rather than decomposing it. Learning Calibrated Medical Image Segmentation via Multi-Rater Agreement Modeling (MRNet) reconstructs each rater's mask and derives a per-pixel uncertainty map from the variance between reconstructions. A Probabilistic U-Net for Segmentation of Ambiguous Images samples spatially coherent hypotheses from a latent space. QUBIQ scores predicted probability maps against rater masks by averaging Dice over ten thresholds. None of these asks at which scale disagreement lives, and none separates positional from structural disagreement. We do not replace these models. We supply a diagnostic and a fusion target that any of them can consume.

Quantitative equational logic supplies a constraint and a correctness argument, not the core mechanism. Quantitative Algebraic Reasoning builds deduction systems for statements $t=_\epsilon s$ and requires operations to be non-expansive. Quantitative Equational Reasoning shows that the Kantorovich (Wasserstein-1) distance arises as the free model of the axioms for weighted mixtures. The non-expansiveness requirement conflicts with blow-up, because a zoom by $1/\lambda$ is arbitrarily expansive. We resolve the conflict by grading distance statements by scale, with judgements of the form $s=^{\lambda}_{\epsilon}t$. A dilation then maps level $\lambda$ to level $c\lambda$ without changing $\epsilon$, which makes it an isometry between graded levels rather than an expansive map. The idea of graded equations follows A Complete V-Equational System for Graded lambda-Calculus. The mixture axioms then give a bound: a fused target $F=\sum_k w_k A_k$ with weights summing to one satisfies $F=^{\lambda}_{\epsilon}A_j$ with $\epsilon\le\sum_k w_k D_{kj}(x_0,\lambda)$.

Research Program

The first component is the measurement tool. At each boundary point of each rater pair, we compute $D_{ij}$ on a logarithmic grid of radii. The grid runs from about three pixels, below which rasterization dominates, to about a third of the object's diameter. We fit the exponent over that window and compute the sign-coherence ratio. Classification thresholds are set on synthetic data and frozen before any clinical analysis. The per-point output is a label (jitter, offset or structure) and a magnitude. Aggregating these per rater also gives spatially varying rater reliability, for example a rater who is consistent on lesion cores but jittery on infiltrative margins. The file flags this spatial variation as unmodeled by current methods.

The second component is the formal layer, kept deliberately small. We will prove soundness of three rules for scale-graded judgements: a graded triangle rule, the dilation rule, and a restriction rule relating nested windows. We will also show that the mixture bound above follows from the barycentric axioms. The purpose is a correctness argument for the fusion constraint, not a new automated prover. Wherever a check is needed in practice, the linear inequalities are solved by linear programming. That is the same algebra that Propositional Logics for the Lawvere Quantale uses for completeness and decidability, via Farkas' lemma and Fourier–Motzkin elimination.

The third component is scale-selective fusion. Starting from DiFF's pipeline, we remove the high-frequency smoother. Instead, at boundary points classified as jitter, we replace the rater masks with their locally averaged boundary. At points classified as structure, we constrain the fusion weights so that the mixture bound keeps the fused target within the structural disagreement envelope, which prevents a notch from being averaged away. This constraint is linear in the weights and is enforced by projection after each update.

Baselines

The naive reference is the per-rater mean soft label, together with majority vote, each trained with soft Dice. This tells us whether any fusion effort matters at all.

The strongest competitor on the fusion question is DiFF with its high-frequency smoother. It is the only method in the file that targets downstream diagnosis and that explicitly addresses spatial noise in fused labels. It reports AUC improvements over consensus baselines of up to 2.73% for glaucoma on REFUGE-2, 2.51% for thyroid cancer on TNMIX and 2.04% for melanoma on ISIC. We will rerun DiFF from its description on identical splits, with the same segmentation and classification backbones, and give it the same hyperparameter budget as our variant. Beating it means a higher downstream AUC than our own DiFF rerun, averaged over at least three seeds, on at least two of the three datasets. On the cases flagged as structural, the fused target must also preserve the measured cup-to-disc ratio or border asymmetry better.

For distribution modeling, the strongest competitors are MRNet and the Probabilistic U-Net. MRNet reports a 1.2% soft Dice improvement for the optic cup on RIGA over prior results. For the Probabilistic U-Net on LIDC-IDRI, no reported result was found on Lacuna for the exact generalized energy distance value. Here we do not claim to beat these models' Dice. We use our profile as an evaluation tool: does each model's sample diversity reproduce the raters' structural disagreement, or only their jitter? The QUBIQ top teams' average Dice of 0.80 to 0.85 serves as a sanity range for our retrained backbones.

Two ablations isolate the key ingredient. The first drops the $1/\lambda$ normalization, measuring disagreement in a fixed-bandwidth Gaussian scale-space of the rater-difference map. The second keeps the normalization but drops the tangent-cone test, using disagreement magnitude alone.

Why not just…?

Why not just use per-point boundary distance statistics between raters? The standard deviation of boundary displacement measures how much raters disagree, not what kind of disagreement it is. A notch drawn by one rater and a uniform two-pixel shift can give the same displacement statistic. The synthetic experiment below tests directly whether displacement statistics alone can separate the three injected types. If they can, our method adds nothing.

Why not just use a classical Gaussian or wavelet scale-space of the disagreement map? This is the real threat. Multiscale analysis is old, and a well-normalized scale-space might behave identically to our blow-up profile. The difference is in the normalization and the readout. Blow-up divides by window area and reads off an exponent, where zero means a different tangent cone. Fixed-bandwidth smoothing instead reports energy per frequency band, where a small notch and jitter can land in the same band. The first ablation is exactly this alternative. If it matches our classifier's accuracy and our downstream AUC, we will report that the contribution reduces to the normalization and the jitter/structure taxonomy, not the blow-up machinery.

Why not just keep DiFF's learned smoother? It works reasonably well by its authors' account. However, it has no mechanism to spare fine but clinically real structure, and DiFF's own visualizations show that consensus-style flattening shifts the cup-to-disc ratio. Head-to-head fusion on REFUGE-2, TNMIX and ISIC at a matched budget will settle whether scale selectivity helps.

Why not skip the logic and just solve the linear constraints? Computationally, we do exactly that. The logic contributes the proof that the graded dilation and mixture bounds are sound, so the fusion constraint is guaranteed rather than heuristic. The logic is not a runtime component. If a reviewer considers the soundness proof routine, the empirical claims stand without it.

Proposed Experiments

The first decisive check is synthetic. We take consensus masks from RIGA and LIDC-IDRI and create artificial raters by injecting controlled jitter, uniform offsets, and small notches or bumps of known size. We then measure how accurately the three types are classified, as a function of feature size relative to image resolution. We report the resolution floor below which structure cannot be told from jitter. This establishes whether the method is usable on fundus images and CT slices at their native resolutions.

The second check is descriptive, on real raters. We apply the profile to RIGA's six raters, LIDC-IDRI's four radiologists, and the QUBIQ tasks with three to seven raters. We report what fraction of disagreement is jitter, offset and structure per task and per rater. We also check whether structural disagreement concentrates where the clinical literature expects it, such as the cup rim and lesion margins. The 3D QUBIQ tasks will be processed slice-wise first, with a full spherical-window version as a stretch goal, since the file notes that 3D uncertainty is where current methods struggle.

The third check is downstream. We compare scale-selective fusion against the mean label, majority vote, and our DiFF rerun on REFUGE-2, TNMIX and ISIC. The measures are classification AUC and the deviation of the fused target's cup-to-disc ratio (or border asymmetry) from the per-rater range, on points the profile labels as structural.

The fourth check is evaluative. We score samples from the Probabilistic U-Net and MRNet by comparing the jitter/offset/structure breakdown of their inter-sample disagreement with that of the raters. We then test whether this comparison ranks models differently from Q-Dice or generalized energy distance.

The proposal fails cleanly in two cases: if the fixed-bandwidth ablation matches our results, or if structural disagreement falls below the resolution floor on these datasets.