Which Disagreements Would More Data Fix? Modality-Aware Annotator Modeling for Material Segmentation

When experts outline regions in an image of a material (where the metal ends and the plastic begins, where one alloy phase gives way to another), they disagree. Medical imaging has built good tools for modeling this disagreement, but they treat it as having one source. Some disagreement, though, exists only because the annotator was looking at the wrong signal. A grey patch in an electron micrograph can be ambiguous, while a chemical map of the same spot would settle it at once. Other disagreement is real: the boundary itself is graded, as in an amorphous-to-crystalline transition, so no added signal will remove it. Materials characterization routinely collects several co-registered probes of the same specimen, such as infrared, polarization, and elemental maps. That makes it the one setting where the two sources can be pulled apart by experiment.

We propose to measure and model this split directly. Annotators segment the same scenes while seeing different subsets of the available modalities. A segmentation model then learns a distribution over each rater's style that depends on which modalities that rater saw. From it we compute, pixel by pixel, how much predicted disagreement would vanish if the missing modalities were shown (reducible) and how much would remain (irreducible). The practical payoff is a map of where an expensive measurement, such as an elemental spectrum, is worth acquiring. The scientific payoff is an empirical answer to an open dispute in the annotator-modeling literature: is human disagreement noise around a single truth, or a set of valid interpretations?

Core Claim

Disagreement among segmentation annotators splits into a part that depends on which modalities they saw and a part that does not. The first part can be predicted from images before the extra modality is collected. Let $A_S(x)$ be the spread of labels at pixel $x$ among raters who saw modality subset $S$, measured as label entropy across raters. Let $M$ be the full modality set. The reducible disagreement is then

$$R_S(x) = A_S(x) - A_M(x),$$

and $A_M(x)$ is the irreducible remainder. We claim three things. First, $R_S$ is large and spatially concentrated on physics-dependent classes and interfaces. Second, a model trained on multi-condition annotations can predict $R_S$ from the cheap modality alone better than generic uncertainty scores can. Third, $A_M$ tracks physically graded transitions, which we measure independently from composition profiles. If this holds, two existing views of annotators become special cases. In the first view, annotators are noisy observers of one hidden truth. In the second, their disagreements are equally valid alternatives. Each view is correct on a different part of the image.

Where This Sits

The multimodal material segmentation work supplies the testbed and the backbone. Multimodal Material Segmentation built MCubeS, a set of road scenes captured in RGB, near-infrared, and two polarization channels with 20 material classes. It showed that the non-RGB channels raise mIoU from 33.7% to 42.86%, with the largest gains on metal, plastic and glass. Multimodal Transformer for Material Segmentation raised this to 53.11% mIoU with modality-specific encoders and a lightweight fusion block. Both papers train against a single deterministic label per pixel, and the first lists uncertainty quantification as future work. We reuse their architecture. What we add is the measurement and modeling of disagreement, which neither attempts.

The annotator-modeling literature supplies the method and the motivating tension. Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation (Pionono) gives each rater a Gaussian in a latent space. The mean of that Gaussian captures the rater's bias and its variance captures their inconsistency. The Latent Doctor Model for Modeling Inter-Observer Variability infers a rater's style from a set of examples that rater has labeled. Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture of Stochastic Experts (MoSE) matches the distribution of predicted masks to the distribution of annotations using an optimal-transport loss. All three condition only on the image and the rater, never on what the rater saw. They therefore cannot say whether a disagreement is curable.

Disentangling Human Error from Ground Truth in Segmentation of Medical Images takes the opposite stance. It assumes one hidden truth and per-annotator confusion matrices. Our decomposition predicts that this stance fits $R_S$ and fails on $A_M$, while MoSE's plurality stance fits $A_M$. We test that prediction rather than choose a side.

Two materials papers constrain the design. A Framework for Supervised and Unsupervised Segmentation and Classification of Materials Microstructure Images uses evidential deep learning, a network that outputs its own confidence, to flag unfamiliar phases for an expert to audit. It treats that expert as infallible and processes each imaging modality separately. A materials data framework and dataset for elastomeric foam impact mitigating materials shows that the subjective threshold used to binarize micro-CT volumes shifts the computed relative density. Together they establish that human judgment in materials segmentation has measurable downstream cost and is currently unmodeled. Hu et al. and Kim et al. supply the kind of graded interface where we expect $A_M$ to be large: amorphous dopant matrices and depth-degraded porous silicon.

Research Program

The work has three parts.

Controlled annotation. On MCubeS we select roughly 150 scenes. Annotators label them under three conditions: RGB only, RGB plus near-infrared, and all four channels. Different annotators are assigned to each condition, so memory of earlier labels cannot leak across conditions. Each scene receives about five labelers per condition. A subset is relabeled by the same people weeks later to measure how consistent individuals are with themselves.

The second dataset is a smaller correlated set from a partner materials lab: backscattered-electron SEM images co-registered with elemental maps from energy-dispersive X-ray spectroscopy (EDS). It covers multiphase specimens that include graded interfaces. Materials scientists segment phases from SEM alone and from SEM plus EDS. Line scans across interfaces give an independent measure of how wide each composition gradient is.

The model. We start from an MMSFormer-style encoder and attach a Pionono-style rater latent. The change is that the latent's distribution is conditioned on the pair (rater, modality subset seen). Training uses MoSE's distribution-matching loss over the annotations within each condition. At test time the network sees only the cheap modality and produces two sets of samples: what raters with subset $S$ would label, and what raters with the full set $M$ would label. The predicted reducible disagreement $\hat R_S$ comes from the difference between the two sets.

The model needs the full modalities at training time only. That is the realistic regime: richer probes exist for a calibration set and are too expensive to collect everywhere.

Use. We turn $\hat R_S$ into an acquisition policy that collects the expensive modality only where predicted reducible disagreement is highest. We also propagate segmentation samples to a downstream physical quantity. For the foam data this is relative density, computed from thresholds that several raters choose independently on the micro-CT volume. The test is whether the predicted spread in density matches the spread across raters.

Baselines

The naive reference is majority vote over all annotators, trained into a deterministic MMSFormer. This is how MCubeS-style models are trained today. It defines the accuracy floor and gives no account of disagreement. For accuracy, the strongest reported number on MCubeS is MMSFormer's 53.11% mIoU (Multimodal Transformer for Material Segmentation). We retrain it on our consensus labels with the same backbone and schedule. Beating it is not the goal; we require that our model's mean prediction stay within one point of it.

For distribution modeling, the strongest competitors are Pionono and MoSE trained on the pooled annotations without modality conditioning. Pionono reports Cohen's κ = 0.758 on Gleason 2019, and the Latent Doctor Model reports KL = 0.461 on Gleason 2019 and a Spearman correlation of up to 0.728 with panel entropy on ISUP. These are on medical datasets. We found no reported result on Lacuna for any multi-rater method on MCubeS or on any materials dataset, so all comparisons will be fresh runs. Every method gets the same encoder, data splits, training budget, and a hyperparameter search of equal size. Beating them means two things. First, lower generalized energy distance (a distance between sets of sampled masks and sets of human masks) to the held-out human distribution in each condition. Second, and decisively, predicting $R_S$ when the unconditioned versions cannot represent it at all.

For predicting $R_S$ from the cheap modality, the competitors are pixel-wise predictive entropy from the unconditioned model, the evidential uncertainty score of A Framework for Supervised and Unsupervised Segmentation and Classification of Materials Microstructure Images, and a deep ensemble's disagreement. Beating them means a higher rank correlation with the measured $R_S$ and a steeper acquisition curve. The key ablation trains our model with the condition label shuffled. Any advantage that survives the shuffle is not coming from modality awareness.

Why not just…?

Why not just show every annotator all modalities and take the vote? This is the most serious threat. If $A_M$ is near zero everywhere, the whole problem reduces to an annotation protocol. Our experiment measures this directly, and a near-zero $A_M$ would be a useful negative result. It still would not remove the modeling problem. Full modalities are usually unavailable at deployment: EDS and XPS maps are slow, and outdoor NIR and polarization channels degrade under extreme lighting (Multimodal Material Segmentation). Deciding where to acquire them requires predicting $R_S$ from cheap data, and this protocol cannot do that.

Why not just threshold the elemental map and skip humans? EDS maps are blurred by the volume of material the electron beam excites, so they are coarser than the SEM image. Choosing their threshold is itself subjective, which is exactly the effect the foam study found for micro-CT (A materials data framework and dataset for elastomeric foam impact mitigating materials). We include threshold-swept EDS labels as an extra "rater" family. If they match the expert distribution under full modalities, this alternative wins for chemistry-defined phases. We would report that and confine our claims to classes that are not defined by chemistry.

Why not just use an existing multi-rater model and read off its entropy? Total entropy mixes curable and incurable disagreement. On a graded interface it is high, yet acquiring more data there buys nothing. The acquisition-curve comparison against entropy and against evidential uncertainty settles this. If entropy performs as well, our conditioning adds nothing and we will say so.

Why not just train one model per condition? Separate models cannot share rater identity across conditions, and they cost several times the compute. They also break down on the materials set, where the annotation pool is small. The Latent Doctor Model reports that a single latent model is competitive with costly ensembles. We run per-condition models as a baseline to confirm this on our data.

Proposed Experiments

The first decisive check is descriptive. On MCubeS, does measured $R_{RGB}$ concentrate on metal, plastic, and glass? Those are the classes whose accuracy rose 9.4–20.9% with polarization (Multimodal Material Segmentation). If disagreement does not drop where accuracy rose, the premise fails.

The second check is predictive. From RGB alone, the predicted $\hat R_{RGB}$ should achieve a clearly higher Spearman correlation with measured $R_{RGB}$ than entropy, evidential, or ensemble scores, and the shuffled-condition ablation should lose that edge. On the SEM-EDS set, the acquisition policy should recover most of the full-modality agreement gain while acquiring EDS over a minority of the area, and its curve should dominate random and entropy-based selection.

The third check is physical. On the SEM-EDS set, irreducible disagreement $A_M$ should correlate with the composition-gradient widths measured independently. We also fit both a confusion-matrix model and MoSE separately on the reducible and irreducible regions. We predict the single-truth model fits the reducible regions better and the plurality model fits the irreducible ones. This gives an empirical reconciliation of the two stances.

Finally, on the foam micro-CT data, the spread of relative density predicted by our samples should match the spread across raters' thresholds more closely than a consensus threshold plus a fixed error bar.