One Separable Network as a Whole Expert Panel

Pathologists often disagree on the same slide, especially on borderline cases. A model trained only on the majority vote learns to sound certain exactly where experts are not. The methods that capture this disagreement well are expensive. One trains a separate model per pathologist and runs all of them. Another samples repeatedly from a learned latent space. Neither fits on the small, low-power devices where a second opinion would be most useful. We propose a cheap way to keep the whole panel. Depthwise separable convolution splits each convolution into a per-channel spatial filter ("where to look") and a 1×1 channel-mixing step ("how to weigh what was seen"). Our hypothesis is that pathologists mostly share the first and differ mainly in the second. We therefore build one network with shared spatial filters and a separate small channel mixer for each pathologist in the last few layers. A single forward pass produces every virtual pathologist's opinion at once.

The bet has two parts. One is scientific: rater disagreement lives in channel mixing, not in spatial filtering. The other is practical: the per-rater mixers can run as one batched matrix multiply on the shared spatial features. That partly offsets the known weakness of depthwise layers on accelerators, which is that they move a lot of memory per unit of arithmetic. If both hold, an edge device gets panel-level calibration for close to single-model cost. If the first does not hold, the matched-parameter ablations will show where rater style actually lives, and that result is useful on its own.

Core Claim

Fix the parameter budget for rater-specific weights. Under that budget, giving each rater its own pointwise (1×1) mixers in the last $k$ separable blocks should reproduce the expert-panel label distribution and calibration better than two alternatives: per-rater classification heads, or per-rater depthwise filters. The resulting model should approach a full independent per-rater ensemble while running in roughly one model's latency on an NVIDIA Jetson Orin Nano Super.

To keep the rater-specific cost sublinear in the number of raters $R$, each rater $r$ gets its pointwise weight in block $\ell$ as a shared matrix plus a low-rank correction:

$$W^{(\ell)}_r = W^{(\ell)}_0 + U^{(\ell)}_r V^{(\ell)\top}_r .$$

Here $W^{(\ell)}_0 \in \mathbb{R}^{N\times M}$ is the shared 1×1 mixer mapping $M$ input channels to $N$ outputs, and $U^{(\ell)}_r \in \mathbb{R}^{N\times q}$ and $V^{(\ell)}_r \in \mathbb{R}^{M\times q}$ are rank-$q$ factors. Training minimizes each rater's own cross-entropy on the cases that rater labeled. At inference the panel distribution is the average of the $R$ softmax outputs. Rater-specific parameters scale as $R\,q\,(M+N)$ per block. The spatial trunk is computed once.

Relation to Prior Work

Three works motivate the problem by showing that keeping rater identity pays off. More is more: virtual expert panel trains one attention-based multiple-instance model per pathologist on Barrett's esophagus biopsies. Its full panel reaches 78.2% average F1 and an expected calibration error (ECE) of 0.050, against 72.9% and 0.183 for a consensus-trained model. Its cost grows linearly with panel size, which is what we aim to remove. The Latent Doctor Model instead learns a continuous latent "doctor style" with a conditional variational autoencoder. It reports KL divergence 0.461 on Gleason2019 and entropy correlation up to 0.728 with 20 pathologists on ISUP. It needs Monte Carlo sampling at inference and is not evaluated on edge hardware. Our model has discrete, named raters and gives all their outputs deterministically in one pass. MRNet reconstructs each rater's segmentation through ConvLSTM modules on a dense U-Net and reports 29 FPS on a desktop GPU. It is the segmentation analogue we compare against on RIGA. DiFF fuses rater masks through a downstream diagnosis network. It addresses a different question, which mask to trust, and requires two full networks.

On the efficiency side, Xception supplies the method: separating spatial and cross-channel correlations costs little accuracy. Network Decoupling shows that pretrained standard convolutions can be converted into separable ones, which gives us a way to initialize the trunk from a standard CNN. Optimizing Depthwise Separable Convolution Operations on GPUs documents the constraint that depthwise layers are limited by memory bandwidth. This is why we place the rater-specific weights in the compute-heavy pointwise step rather than the depthwise step. LitePath is the closest edge-pathology system. It distills large pathology models into a 22.5M-parameter ViT-Small and processes 208 slides per hour on the Orin Nano Super. It treats labels as single ground truth and notes that once compute is cut, data loading becomes the bottleneck. We reuse its hardware target and treat its student as a competing backbone. Edge detectors such as Tiny SSD, with 1.13M parameters and 2.3 MB, show the footprint range that factorized microarchitectures reach. None of them model annotators. The one separable-CNN medical work in the file, on Alzheimer's detection, uses single labels and standard hardware.

Research Program

We build a MobileNet-style separable trunk, with inverted residual blocks, initialized from ImageNet weights or decoupled from a pretrained ResNet. Rater-specific low-rank pointwise mixers are attached in the last $k$ blocks. On segmentation the same design applies inside a separable U-Net decoder, with rater mixers in the decoder's final blocks.

The central analysis is a placement study at a fixed rater-specific parameter budget. Rater-specific capacity goes either into depthwise filters, into pointwise mixers, or into classification heads only. We also sweep the depth $k$ and the rank $q$. If disagreement is a matter of weighting shared morphological evidence, the pointwise placement should win and the gain should saturate at small $k$. If it is spatial, for example different attention to chromatin texture, the depthwise placement should win. Two further tests check that the model encodes real rater style. In the first we hold out a rater's labels and predict them, which measures fidelity to individual raters. In the second we fit a new rater from a small support set by training only that rater's $U, V$ factors.

The engineering piece implements the $R$ mixers as one stacked pointwise convolution with output width $R\cdot q$, followed by per-rater projections. We profile latency, peak memory and energy on the Orin Nano Super in FP16, so that hardware cost is measured rather than inferred from parameter counts.

Baselines

The naive reference is the same separable trunk trained on majority-vote labels. This is what a typical edge deployment would ship, and it tells us how much calibration consensus training loses on these datasets. On DEPP the consensus model's reported ECE is 0.183 (virtual expert panel).

The strongest cheap competitor is the same trunk trained with soft labels, that is, cross-entropy against each case's empirical rater distribution. It costs nothing extra at inference and directly targets the panel distribution. It is run with identical data, schedule and hyperparameter search budget. We count it as beaten only if our model wins on KL and ECE across seeds by more than the seed-to-seed spread.

The key ablation is the trunk with per-rater linear heads, a standard multi-head design. It is given exactly the same number of rater-specific parameters as our mixers. This baseline isolates the core ingredient: whether rater-specific channel mixing inside the network adds anything over rater-specific readout.

The quality ceiling is $R$ independent copies of the separable network, one per rater. This is a small-scale version of the full virtual panel. We expect to approach it, not beat it, and we report what fraction of its KL and ECE advantage over soft labels we recover per unit of latency. A five-member deep ensemble of the trunk is also run, mirroring the ensembling result in MIDOG, where five models reached 0.773 F1 against 0.748 for the best single model.

The strongest existing multi-rater method is the conditional Latent Doctor Model. Its reported Gleason2019 KL divergence is 0.461 and its ISUP entropy Spearman correlation is up to 0.728 (LDM). We rerun it on our splits, both with its original backbone and with our separable trunk, and with matched training compute. Beating it means equal or lower KL with at least several times lower edge latency than its sampling loop. Matching KL within seed noise at much lower cost also counts as a win for the deployment claim.

On RIGA the competitor is MRNet. Its reported result is a 1.2% optic-cup soft-Dice gain at 29 FPS on a workstation GPU. We run it and our model on the same split and both devices. For 3k Buds no reported numeric result was found on Lacuna, so all methods there are run fresh.

Why not just…?

Why not just train a small separable model on soft labels? This is the most serious threat, because soft labels already encode the panel distribution at zero inference cost. What soft labels discard is which rater said what. That matters for predicting a specific held-out rater, for adapting to a new rater, and possibly for calibration on cases where raters disagree systematically rather than randomly. The virtual-panel paper's gain over consensus does not by itself show a gain over soft labels. If our model ties soft labels on KL and ECE, the deployment recommendation becomes "use soft labels." The placement study would then still report where rater style lives, which is useful for rater adaptation.

Why not just put per-rater heads on a shared backbone? Heads are cheaper still, and if raters differ only in their decision thresholds, heads are enough. The matched-parameter ablation answers this directly. We predict mixers win most on ordinal grading with many raters, such as ISUP's 20 pathologists and DEPP-like grading. There, raters plausibly weigh different features rather than just shifting thresholds. If heads tie, the core claim is falsified.

Why not just use a distilled transformer student like LitePath? LitePath keeps 99.71% of Virchow2's AUC on the edge, so a strong backbone is available. However, it has 22.5M parameters, its distillation used about 190M patches, and it has no rater modeling. Its per-rater heads would face the same head-versus-mixer question. GasHisSDB found a plain ViT trailing CNNs (94.59% against 96.47%) and needing 400–500 epochs, though that is weak evidence against a distilled student. If LiteFM weights are accessible, we run LiteFM features with per-rater heads as a backbone-strength control. Separately, we apply low-rank rater mixers to its MLP layers to test whether the idea transfers beyond convolutions.

Why not just ensemble a few tiny models? Ensembles capture model uncertainty, not rater identity, and their cost grows with ensemble size. We run the five-member ensemble at matched total parameters and report KL, ECE and edge latency side by side.

Proposed Experiments

The classification experiments use three datasets. Gleason2019 has 244 cores, 6 pathologists and 4 classes. 3k Buds has 3,000 candidates, 9 pathologists and 3 classes. ISUP has 100 slides and 20 pathologists; there we use patch-level training with slide-level aggregation and treat it mainly as an uncertainty-correlation test given its size. Segmentation uses RIGA, with 750 fundus images and 6 experts. If DEPP access is obtainable, we evaluate the mixer idea on its patch encoder. Otherwise DEPP figures serve only as context.

Metrics follow the cited work. Distributional quality is measured by KL to the empirical rater distribution, ECE, and Spearman correlation between model and panel entropy. Individual-rater fidelity is accuracy on held-out raters' labels. Task performance is majority-vote F1 or soft Dice. Deployment cost is FP16 latency, peak memory, energy per image and achieved arithmetic intensity on the Jetson Orin Nano Super. Every comparison uses five seeds.

The decisive checks are three. First, at matched rater-specific parameters, pointwise mixers must beat per-rater heads and soft labels on KL or ECE on at least two datasets; otherwise the central claim fails. Second, adding the $R$ mixers must raise edge latency by well under the factor of $R$ paid by the independent panel, with profiling showing that the batched pointwise step is compute-bound. Third, the depthwise-placement ablation tells us whether disagreement is spatial; a depthwise win would overturn our hypothesis and redirect the design. A secondary test checks robustness under shifted inputs. Following MIDOG's finding that scanner shift inflates false positives on look-alike cells, we test on unseen-scanner splits whether the predicted panel entropy rises on shifted inputs instead of collapsing into confident errors.