How Much of a Coating's Reported Benefit Depends on Who Traced the Bone?

Bone-implant studies judge a surface coating by how much new bone grows against the metal. That quantity is measured by a person who traces bone in micro-CT scans or stained tissue sections. Take polydopamine, a mussel-inspired "molecular glue" that sticks to almost any surface and holds peptides, drugs or nanoparticles. A polydopamine-coated titanium implant is typically declared better because traced bone-to-implant contact (BIC, the fraction of implant surface touching bone) and bone volume fraction went up. The tracing itself is treated as ground truth, yet it happens exactly where images are least trustworthy. Metal causes streaking artifacts in CT, and newly mineralizing bone fades gradually into soft tissue. This proposal treats the traced outcome as a random variable that depends on the rater. We collect many experts' tracings of coated and uncoated implants, and fit a model of how individual raters behave. Pushing that model's rater population through to the final comparison gives each coating effect an interval that honestly includes annotation disagreement.

The sharper hypothesis goes beyond "tracing is noisy." We claim the noise is not neutral between study arms. A coating changes the image near the interface: it adds a thin organic layer, alters how mineral density ramps up, and changes staining. So raters may disagree differently, and lean differently, on coated versus uncoated implants. When the uncoated baseline is small, as in ratios like the 1409% BIC gain reported for a DOPA-peptide titanium coating in Peptide-based inflammation-responsive implant coating sequentially regulates bone regeneration to enhance interfacial osseointegration, a boundary shift of a few voxels can move the headline number a great deal. We do not claim that result is wrong. We ask whether the field has a way to know how fragile such numbers are, and we build one.

Core Claim

Our claim is that, in implant imaging, the comparison between arms can be sensitive to rater choice. Rater bias can interact with coating status, and standard reproducibility statistics computed per image will miss this. A rater-style segmentation model gives the missing tool. Such a model learns each annotator's systematic lean and inconsistency, and can sample plausible new annotators. With it, one can report an annotation-aware treatment effect with intervals, and flag which animals and slices drive the fragility. What is new is where the uncertainty is carried: into the group-level biomaterial comparison, with an independent mechanical measurement as the check. Existing work carries it only as far as pixel-level segmentation metrics.

Relation to Prior Work

Four latent-variable methods model expert disagreement. Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation, called Pionono, gives each rater a Gaussian in an 8-dimensional latent space inside a U-Net (an encoder-decoder segmentation network). The Gaussian's mean encodes systematic bias and its variance encodes inconsistency. The Latent Doctor Model for Modeling Inter-Observer Variability infers a doctor's style from a small set of that doctor's labels. Variational Inference for Quantifying Inter-observer Variability in Segmentation of Anatomical Structures samples plausible segmentations from a 6-dimensional latent space. Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle weights experts by how well their masks support a downstream diagnosis. These papers supply our method. All are evaluated only on clinical MRI, CT, fundus and pathology images. All stop at segmentation or diagnosis metrics and never ask how a between-group scientific conclusion changes.

Two papers motivate the problem. The DOPA-peptide implant study reports BIC and bone volume from micro-CT and Van Gieson histology without reporting rater variability. Variability Matters: Evaluating Inter-Rater Variability in Histopathology for Robust Cell Detection found Fleiss' kappa of only 0.43 among 120 pathologists, which shows that expert tracing at ambiguous boundaries can agree poorly.

Two findings constrain the design. The first is the QUBIQ organizers' observation that plain ensembles of deterministic networks often matched complex probabilistic models (QUBIQ). We therefore run an ensemble as a serious competitor, not a strawman. The second is the diagnosis-first idea, which we deliberately do not use as a training target. Push-out force, the force needed to dislodge the implant, is measured once per animal on a few dozen animals. Learning expert weights from it would overfit. We keep it as held-out validation only.

Research Program

The work has three parts.

Build a dataset. We will assemble a multi-rater implant dataset. In partnership with a lab running rat femoral titanium implants, we take polydopamine-primed peptide-coated implants and uncoated controls at two healing times. For each animal we collect micro-CT volumes, undecalcified Van Gieson sections and push-out force. Between eight and twelve raters trace bone within a fixed peri-implant band: bone biologists, orthopedic researchers and micro-CT technicians. Each rater retraces a random subset weeks later, so that within-rater inconsistency can be estimated. Half of each rater's cases are shown blinded to arm and half unblinded, in a crossover design. This separates two sources of bias: bias from knowing the hypothesis, and bias driven by what the image looks like. No public dataset of this kind is listed on Lacuna.

Fit rater-style models. We fit Pionono-style models on the dataset. Before touching implant data, we check our implementation on Gleason 2019 and QUBIQ against the published numbers.

Estimate the treatment effect under rater uncertainty. For a rater latent $z$, let $\Delta(z)$ be the difference between arms in mean predicted BIC, computed over animals. Here $z$ is drawn either from a fitted rater's distribution or from the population prior over raters. The annotation-aware effect is the distribution

$$ \Delta(z) = \frac{1}{n_c}\sum_{i \in C} \mathrm{BIC}(x_i; z) - \frac{1}{n_u}\sum_{j \in U} \mathrm{BIC}(x_j; z), \quad z \sim p(z), $$

where $C$ and $U$ are the coated and uncoated animals, $n_c$ and $n_u$ their counts, and $x$ an animal's image stack. We combine this with ordinary between-animal resampling, which yields an interval that covers both biological and annotation uncertainty. We also fit a mixed model to the raw human tracings, with a rater-by-arm interaction term. That term answers the central question without any neural network at all.

Baselines

The naive reference point is current practice. One trained operator traces with a threshold plus manual region cleanup, and the effect is reported with between-animal confidence intervals only. The proposal beats this reference if its intervals cover the effect estimates of held-out human raters that the single-operator interval misses.

The classical statistical baseline uses multiple raters with an intraclass correlation. Their masks are fused by per-pixel mean or by STAPLE, an expectation-maximization method that estimates a consensus mask while weighting each rater's reliability. The effect is computed on the fused mask. Every model sees exactly this multi-rater data.

The deterministic automation baseline is a single fixed global threshold with fixed artifact-exclusion rules, applied identically to all scans. We also sweep that threshold over the range raters actually choose, which shows how much of the variability is simply a threshold choice.

The strongest learned competitor is a deep ensemble with one U-Net per rater, which QUBIQ found hard to beat. It gets the same backbone, augmentation and total GPU-hour budget as Pionono.

The strongest existing method is Pionono itself. On Gleason 2019 it reports unweighted Cohen's kappa of 0.758, against 0.750 for the prior method (Pionono). For distribution matching, the Conditional Latent Doctor Model reports a KL divergence of 0.461 to the expert panel on Gleason 2019 (LDM). On QUBIQ brain growth, variational inference reports a continuous Dice of 0.5372, against 0.5083 for Bayes Dropout (VI). There is no reported result on Lacuna for any multi-rater model on implant micro-CT or histology.

The key ablation removes the step that carries uncertainty to the group level. It uses the same trained Pionono but reports the effect under the "gold" consensus latent only. If that interval is as good as the full propagated one, our central ingredient adds nothing.

On implant data, "beating" a baseline means three things. The held-out-rater effect estimates must fall inside the nominal 90% interval about 90% of the time. The model must have lower KL to the held-out raters' per-pixel distribution. And fragility flags must predict where raters actually disagree.

Why not just…?

Why not just have three people trace and report an intraclass correlation? An intraclass correlation summarizes agreement across images, while the scientific quantity is a difference between arms. If raters lean the same direction on both arms, agreement is high and the effect is unaffected. If they lean differently by arm, agreement can still look acceptable while the effect shifts. The mixed-model interaction term on raw tracings is exactly this simpler analysis, and we run it first. If it shows no rater-by-arm interaction and small rater variance, the neural propagation is unnecessary for this system, and we will say so. That would itself be a useful negative result for the field.

Why not just automate with a fixed threshold, since a deterministic rule has no inter-observer variability? A fixed threshold makes variability invisible, not absent. Its value is one arbitrary point in the range experts would choose. If a coating shifts the mineral-density profile near the interface, the threshold's bias can differ by arm, just as a human's can. Our threshold sweep tests whether the conclusion holds across the thresholds raters actually use. If it does, the threshold is an adequate, cheaper answer.

Why not just blind the raters? Blinding removes bias that comes from knowing the hypothesis. It does not remove bias caused by coated interfaces looking different. The blinded/unblinded crossover measures both. The arm-dependent bias that remains among blinded raters is the part that only modeling can expose.

Why not just use a per-rater deep ensemble? It is a real threat on accuracy, and QUBIQ reports top teams reaching average Dice of 0.80 to 0.85 largely with ensembles. The ensemble can represent only the raters it was trained on. It cannot sample a new rater from a population, and it is what one needs to state an interval over "a different lab's operator." We compare the two on coverage for held-out raters at matched compute. If the ensemble's empirical spread covers as well, we adopt it and drop the latent model.

Decisive Checks

Three results decide the question. The first is the size and sign of the rater-by-arm interaction in the blinded half of the human data. The second is whether the width of the annotation-aware interval for BIC and bone volume fraction is comparable to between-animal uncertainty, or negligible beside it. The third is a check against mechanics. If rater variability is real, rater-specific BIC should correlate with push-out force to different degrees across raters, measured by per-rater Spearman correlation across animals. The model's population-averaged BIC should correlate at least as well as the best single rater. The project succeeds even if the answer is "tracing is robust here," provided the check is rigorous and reusable for other polydopamine-functionalized interfaces.