Porous nanoparticles, such as mesoporous silica, hollow silicon/carbon shells and honeycomb polymer networks, owe much of their function to their geometry: pore width, shell thickness, and how open the pore entrances are. Those quantities are still routinely read off electron micrographs by people tracing boundaries by hand. That is the same subjective, expert-dependent judgment that medical imaging has spent a decade learning to model. Machine-learning methods for multi-rater data (models that learn each annotator's habits instead of averaging them away) can reproduce how experts disagree. They have one blind spot that medical data hides. If every expert shares the same bias, the model learns that bias as "truth," and in pathology nothing independent exists to catch it.
Nanomaterials offer that independent check. A gas-adsorption measurement, which infers pore sizes from how much nitrogen a powder absorbs, gives an instrument-derived pore-size distribution for a whole batch. A synthetic micrograph rendered from a known 3D pore model gives exact truth. We propose to build the first multi-rater annotation benchmark for porous-nanoparticle electron micrographs. On it we will test a specific claim: a rater-latent model that also treats the physical instrument as one more "rater," with a known noise model at the batch level, recovers bias that all humans share. Unanchored multi-rater models cannot do this, and neither can classical automatic segmentation.
Core Claim
Current multi-rater models separate each annotator's systematic offset from their inconsistency. They cannot separate the bias the whole panel shares from the truth, because nothing in the data identifies it. Pionono embeds each rater as a Gaussian in an 8-dimensional latent space, with the mean as personal bias and the covariance as inconsistency. Its "rater-free" prediction is still centered on the panel. The Latent Doctor Model uses a conditional variational autoencoder to reproduce the distribution of expert labels, and it scores this by KL divergence to the panel. Both papers motivate our work, since they show rater structure is learnable. Both also define success relative to humans. We claim that adding a weak physical observation, available only as a batch-level histogram and not per pixel, makes shared human bias measurable and correctable. We also claim this matters for material function, because image-derived geometry is used to explain loading and accessibility.
The modeling change is small. Take a Pionono-style segmentation model with per-rater latents. Add a learned "true" latent whose predictions, pooled over all particles in a synthesis batch $b$ and passed through a fixed projection correction, must match the instrument histogram:
$$\mathcal{L} = \sum_{r}\sum_{i} \mathcal{L}^{\text{rater}}_{r,i} + \lambda \sum_{b} D\big(g(\hat{H}_b),\, H^{\text{phys}}_b\big)$$
Here $\mathcal{L}^{\text{rater}}_{r,i}$ is the usual per-rater evidence-lower-bound term for annotator $r$ on image $i$. $\hat{H}_b$ is the pore-size histogram the model predicts for batch $b$ under the true latent. $g$ is the correction from 2D projected widths to the adsorption-equivalent width. $H^{\text{phys}}_b$ is the measured histogram. $D$ is a Wasserstein distance, and $\lambda$ sets the anchor's weight. The per-rater terms still explain how each person deviates from the true latent, so their shared offset becomes an estimated quantity rather than an absorbed one.
Research Program
The program has three parts. The first is a dataset. We will collect transmission and scanning electron micrographs of three architecture families described in the file. The first family is mesoporous silica with calibrated pores, as in Zou et al., who used high internal surface area for analyte capture. The second is hollow mesoporous Si/C shells, as in Zhang et al., whose hollow core gave drug loading above 30% of carrier mass. The third is honeycomb molecularly imprinted polymers, as in Guo et al.. For each batch we will record nitrogen-adsorption pore-size distributions and, where possible, measured payload loading. Eight to twelve materials scientists of mixed seniority will independently trace particle outlines, shell boundaries and pore openings. A subset of images will be traced twice by the same person to estimate within-rater inconsistency.
The second part is a synthetic twin set. We will render micrographs from procedurally generated 3D pore networks and hollow shells, with realistic projection, defocus and shot noise. The same raters will annotate a sample of these. Here truth is exact, so shared human bias, for example missing pores near the resolution limit, can be measured directly rather than inferred.
The third part is the anchored model above, plus analyses of where human disagreement concentrates. A domain detail makes this worth doing: work on electrostatic barriers to pore access shows that high internal area does not guarantee entry, because aperture geometry and charge gate access. Aperture width is exactly the small, ambiguous feature where tracing is most subjective. Batch-average adsorption data cannot report it per particle.
Baselines
The naive reference point is a standard U-Net trained on the pixel-wise majority vote of the panel. In the medical setting this is known to be overconfident: on Barrett's esophagus grading, the virtual expert panel study reports a consensus-trained model at ECE 0.183 and average F1 72.9%, against 0.050 and 78.2% for their full panel of per-rater networks. We include majority vote because it is what a materials lab would actually do. If the anchored model does not beat it on recovering truth, the whole program fails.
The strongest existing method is Pionono itself. It reports unweighted Cohen's kappa 0.758 on Gleason2019, over a previous best of 0.750, and 0.711 on a 25-rater breast segmentation set. We will first reproduce it on Gleason2019 to within a small margin of 0.758, so that any later gap is not an implementation artifact. We then train it on our micrographs with the same backbone (ResNet34 U-Net), the same data splits, the same compute and the same hyperparameter-search budget as our model. The virtual expert panel and the Latent Doctor Model, which reports KL 0.461 on Gleason2019, serve as additional distribution-learning baselines under the same budget. For porous-nanoparticle micrographs specifically, there is no reported result found on Lacuna for any multi-rater method, so every number on the new benchmark will be ours.
A classical automatic pipeline is a separate competitor: thresholding plus watershed, with parameters tuned on the training split against majority vote. It also gets a second tuning against the adsorption histograms, so it has access to the same physics. The key ablation is our model with the anchor replaced by a batch-shuffled physical histogram. This keeps the regularization strength while destroying the information. If the shuffled anchor helps as much as the true one, the gain is regularization and not physics.
We will count it as beating the baselines if three things hold. First, on synthetic data, the anchored model's error in recovering the true pore-size distribution is clearly below Pionono's and majority vote's, with batch-level bootstrap intervals that do not overlap. Second, on held-out real batches, it predicts measured adsorption statistics better. Third, it matches Pionono on per-rater fidelity (kappa against held-out individual raters), so it does not pay for truth with a worse model of the people.
Why not just…?
Why not just average the raters, or use Pionono as is? Averaging and Pionono both converge to the panel's center, so any bias the whole panel shares passes straight through. The synthetic set settles this directly. If humans show no shared bias there, the anchor has nothing to fix, and we will report that the existing methods suffice for this domain.
Why not just replace humans with automatic segmentation? This is a real threat. A tuned watershed has no inter-observer variability at all. Its thresholds, however, are one fixed "rater" with its own bias, and it does not report where it is uncertain. We give it the physics anchor during tuning, so the comparison isolates whether learning human rater structure adds anything. If the tuned classical pipeline matches the anchored model on held-out adsorption agreement and on synthetic truth, we concede that the human-modeling half is unnecessary for metrology. The rater analysis would then remain useful only as a measure of how biased manual practice has been.
Why not just use gas adsorption and skip the images? Adsorption reports one batch average. It cannot give per-particle heterogeneity, shell thickness, or aperture width, which the electrostatic-barrier work shows governs access. We test whether image-derived per-particle aperture statistics from the anchored model predict measured loading better than adsorption averages alone. Loading is the property Zhang et al. rely on. If they do not predict it better, the images add nothing beyond the instrument.
Proposed Experiments and Decisive Checks
The first decisive check runs on the synthetic twins. We estimate the panel's shared bias, as a function of pore size and defocus, from the human annotations. We then show that unanchored Pionono inherits that bias and the anchored model removes most of it. If humans are unbiased on synthetic data, the central claim is false.
The second check holds out entire synthesis batches. We compare each method's predicted pore-size distribution and surface area with the measured adsorption values. We also run the shuffled-anchor ablation to rule out pure regularization.
The third check asks whether per-particle predictive spread tracks human disagreement. We measure this with the Spearman correlation between model entropy and panel entropy, following the protocol on which the Latent Doctor Model reports up to 0.728 on 20-pathologist prostate grading. This tells us whether anchoring preserves the model of disagreement while correcting its center.
The final check is functional. In a regression on held-out batches, do anchored image-derived aperture statistics explain measured payload loading better than majority-vote tracings or adsorption averages? A positive answer would mean that modeling observer bias changes what a materials scientist concludes about structure and function, not just a segmentation score.
References
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- The Latent Doctor Model for Modeling Inter-Observer Variability
- A porous nano-adsorbent with dual functional groups for selective binding proteins with a low detection limit
- Degradable Hollow Mesoporous Silicon/Carbon Nanoparticles for Photoacoustic Imaging-Guided Highly Effective Chemo-Thermal Tumor Therapy in Vitro and in Vivo
- Preparation and Application of MolecularlyImprinted Polymers for FivePyrethroid Insecticides
- Electrostatic Barriers to Nanoparticle Accessibility of a Porous Matrix
- More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel