Mass-Balance Label Purification for Hyperspectral Mycotoxin Screening

Hyperspectral cameras can photograph each almond or wheat kernel across hundreds of narrow wavelengths, and the chemical fingerprint of fungal toxins such as aflatoxin B1 shows up faintly in those spectra. Training a kernel-level toxin detector needs kernel-level labels. The chemistry that supplies ground truth cannot provide them. High-performance liquid chromatography (HPLC), the reference assay, grinds a whole lot of kernels into powder and reports one average concentration. Toxin accumulates in a few "hot" kernels, so copying that lot average onto every kernel in the lot mislabels most of them. The field has sidestepped this by pipetting uniform toxin droplets onto clean kernels, which gives clean labels but not natural contamination.

We propose a cleaning step that runs before training and converts lot-level chemistry into trustworthy kernel-level labels. It combines two signals. The first is spectral neighborhoods: a kernel whose spectrum resembles known-clean kernels is probably clean. The second is a mass-balance constraint: the kernel concentrations inside a lot must average to the value HPLC measured. The output is a relabeled dataset that any standard chemometric classifier, such as a support vector machine (SVM) or partial least squares, can train on unchanged. The cleaner is also designed so that it does not throw away the hard positive kernels, the ones that look clean but are not, which are exactly the kernels a food-safety screen exists to catch.

Core Claim

Label noise in mycotoxin spectroscopy is not random flipping. It has a known structure: each kernel carries its lot's label, and the lot's true average concentration is measured. We claim that a cleaner which uses both spectral similarity and this aggregate constraint will recover kernel labels far better than generic noise filters, which assume unstructured noise. We also claim that doing this cleaning with an accuracy-driven objective, and applying asymmetric regulatory costs only afterwards, will lower the rate of missed contaminated kernels at a fixed false-alarm rate.

Concretely, lot $b$ contains kernels $i$ with masses $m_i$ and has a measured bulk concentration $c_b$ with assay uncertainty $\sigma_b$. A neighborhood model assigns each kernel a prior score $s_i \in [0,1]$ for exceeding a regulatory threshold. The cleaner then solves, per lot:

$$\min_{q}\ \sum_{i\in b} \mathrm{KL}\big(q_i \,\|\, s_i\big)\quad \text{s.t.}\quad \Big|\tfrac{\sum_{i\in b} m_i\, \hat z(q_i)}{\sum_{i\in b} m_i} - c_b\Big| \le \sigma_b .$$

Here $q_i$ is the cleaned soft label, and $\hat z(q_i)$ is the concentration implied by that label, calibrated on the pipetted-standard data. The constraint lets lots measured near zero act as nearly noise-free negative anchors. Contaminated lots can then place their toxin mass only on the few kernels whose spectra support it.

Relation to Prior Work

Two papers are the closest bridges between noisy-label cleaning and spectral agriculture. Neither addresses aggregate chemical labels.

Hyperspectral Image Classification in the Presence of Noisy Labels cleans hyperspectral training labels before training. It propagates labels inside spatially contiguous superpixels (small regions of similar-looking pixels). On Indian Pines with 50% flip noise, this raised an extreme learning machine's accuracy from 67.86% to 86.78%. This paper motivates our work, because it shows that cleaning before training pays off on spectral data. It also constrains our design. Its assumptions are spatial homogeneity and uniform random noise, and both fail for kernel-scale hot spots whose noise is set by lot membership. We therefore replace spatial smoothing with a mass-balance constraint.

Rectified Meta-Learning from Noisy Labels for Robust Image-based Plant Disease Diagnosis handles noisy plant-disease labels inside training, using a nearest-neighbor meta-learning loop on RGB leaf photos. On PlantVillage at 90% symmetric noise it reaches 64.41% accuracy, versus 38.36% for plain cross-entropy. We differ in three ways: we clean once before training, we work on spectra rather than visible lesions, and we exploit a measured aggregate that the leaf setting lacks.

Our method borrows from two lines of work that clean labels without training a predictor. Detecting Corrupted Labels Without Training a Model to Predict (SimiFeat) flags a sample whose label disagrees with the vote of its nearest neighbors in feature space. Candidate Label Set Pruning uses neighbor votes to prune candidate labels, improving 145 of 149 partial-label settings. Our prior score $s_i$ is a SimiFeat-style neighbor vote. The mass-balance projection is what these methods lack. Without it, a neighbor vote over lot labels inherits the lot's bias, since most neighbors of a kernel from a contaminated lot are also labeled contaminated.

Two further papers constrain the design. An Empirical Study of the Noise Impact on Cost-Sensitive Learning shows that learners optimized for asymmetric costs collapse toward the cheap class under label noise, and that cleaning should use accuracy-driven models. We follow this: costs enter only at training and thresholding, never in the cleaner. DynClean removes samples with low margin during training. The walnut study Non-Destructive Detection of Moldy Walnuts Based on Hyperspectral Imaging Technology found internally moldy, externally normal walnuts to be the hardest class, at 78.6% accuracy. Margin-based pruning would plausibly discard exactly these kernels. For that reason our cleaner relabels softly rather than deleting, and we test the margin-pruning failure directly.

Research Program

The work has three parts.

First, we build a controlled benchmark in which kernel-level truth is known. The Nonpareil almond aflatoxin B1 dataset from Deep Learning-Based Detection of Aflatoxin B1 Contamination in Almonds Using Hyperspectral Imaging has 3,596 kernels labeled at 0 to 1000 ppb, with an HPLC-verified subset. The morphology-varied almond cohorts from Multilayer Perception-Based Hybrid Spectral Band Selection Algorithm for Aflatoxin B1 Detection Using Hyperspectral Imaging add physical variation. We will obtain these datasets or re-collect them under the same protocols. From them we form synthetic lots by sampling kernels, and we compute each lot's mass-weighted concentration plus simulated assay error. Three factors will be varied: lot size, prevalence of hot kernels, and whether lots are drawn from the uniform-shape cohort or the variable-shape cohort (Dataset 3). The shape comparison tests the cleaner where spectral neighborhoods are least reliable.

Second, we implement the cleaner. It has two stages: neighbor scoring in a standard-normal-variate plus principal-component spectral space (optionally in a small learned 1D-CNN embedding), followed by the per-lot constrained projection above. The projection is a small convex problem, cheap even for large lots.

Third, we collect a modest naturally contaminated validation set. It consists of almond lots that are imaged kernel by kernel, with a random kernel subset assayed individually by HPLC before the remainder is pooled. This set is the only real test that synthetic lots are not flattering the method.

Baselines

The naive reference assigns every kernel its lot's label, trains the same SVM and the same 3D CNN, and applies the regulatory threshold. This is what a practitioner with bulk chemistry would do today.

The strongest detection reference is the lightweight 3D Inception-ResNet trained on clean kernel labels. It reports 90.81% validation accuracy, F1 0.899 and AUC 0.964 on pipetted almonds Deep Learning-Based Detection of Aflatoxin B1 Contamination in Almonds Using Hyperspectral Imaging. A second reference is MLPW-SR with an SVM, which reports 98.67% cross-validation accuracy and F1 0.982 on four bands of uniform-shape almonds Multilayer Perception-Based Hybrid Spectral Band Selection Algorithm for Aflatoxin B1 Detection Using Hyperspectral Imaging. These numbers are ceilings: they were obtained with oracle kernel labels. Success means closing most of the gap between the naive reference and this oracle when training from lot labels.

The strongest cleaning baselines are SimiFeat voting applied to lot labels, AUM-margin pruning in the style of DynClean, and random label propagation from Hyperspectral Image Classification in the Presence of Noisy Labels. The last is adapted by treating each lot as a group in place of its superpixels. Its reported gain is on land-cover data at 50% noise (67.86% to 86.78%). No reported result was found on Lacuna for any label cleaner applied to mycotoxin spectra, so all cleaning comparisons will be run fresh.

Every cleaner receives the same spectra, the same preprocessing, the same downstream classifier, and the same tuning budget. Tuning uses a held-out set of lots selected only by lot-level agreement, with no kernel oracle. Ablations remove the mass-balance constraint (plain neighbor voting) and remove the neighbor prior (constraint with a uniform prior), which isolates which ingredient does the work.

Beating the baselines means lower kernel-level false-negative rate at a matched false-positive rate, and higher label-recovery F1, across at least the variable-shape cohort and the natural validation set.

Why not just…?

Why not just train end-to-end with multiple-instance or label-proportion learning? Multiple-instance learning trains a model from labels on groups ("bags") rather than individuals, and label-proportion learning trains from the known fraction of positives in each group. Both handle exactly this lot-to-kernel ambiguity, so this is the real threat. The file contains no reported result for either on mycotoxin spectra. We will implement a proportion-loss 1D-CNN and an attention-based multiple-instance model on identical lots. If either matches our cleaner in false-negative rate, the remaining contribution is portability: the cleaned labels feed the SVM and partial-least-squares tools regulators already validate. We would report that narrower claim honestly. We expect the cleaner to help most with small lots and few lots, where end-to-end models have little signal to fit.

Why not just apply a cost-sensitive classifier directly to lot labels? An Empirical Study of the Noise Impact on Cost-Sensitive Learning shows such learners saturate toward the cheap class as noise rises. We will reproduce this with a cost-weighted SVM on naive lot labels across cost ratios, and compare it with the same learner trained on cleaned labels.

Why not just rely on chemometric preprocessing and band selection? Scatter correction and band selection fix noise in the spectra, not wrong targets. Walnut internal-mold accuracy stayed at 78.6% despite this preprocessing Non-Destructive Detection of Moldy Walnuts Based on Hyperspectral Imaging Technology. All our arms use this preprocessing, so any gain comes from label correction alone.

Why not just prune low-margin samples as DynClean does? Pruning removes hard examples. The decisive measurement is the fraction of truly contaminated kernels that each cleaner deletes or flips to clean. If margin pruning does not disproportionately remove hard positives, this part of our motivation fails.

Proposed Experiments

The main experiment crosses cleaners with downstream classifiers on synthetic lots. Lot sizes range from 5 to 50 kernels, and hot-kernel prevalence ranges from 2% to 30%. We report three quantities: label-recovery F1 against kernel truth, kernel-level false-negative rate at 5% false positives, and a lot-level sorting metric. The sorting metric is the residual mass-weighted concentration of a lot after the kernels the classifier flags are removed, which is the operational goal of sorting.

The first decisive check is the constraint ablation. If plain neighbor voting on lot labels matches the full method, the mass-balance idea adds nothing.

The second decisive check is the variable-shape cohort. If spectral neighborhoods collapse there, both the neighbor-only and full methods should degrade. The constraint should degrade more gracefully, because clean-lot anchors remain informative.

The third decisive check is the naturally contaminated validation set. Synthetic lots built from pipetted kernels may make hot spots easier to find than natural fungal infection does. A gain that disappears on natural kernels would mean the benchmark is flattering the method.