When several radiologists outline the same tumor, they draw different boundaries. The disagreement is clinically real: it changes radiation doses and makes downstream measurements unstable. Models that reproduce each expert's style, rather than a single averaged mask, give clinicians a view of where the boundary is genuinely uncertain. The strongest such models are ensembles or networks with one decoder per expert. They cost a full or partial forward pass per expert, which is why the QUBIQ challenge organizers single out compute as the barrier to clinical use (QUBIQ). Cheap per-rater adaptation exists in the form of PU-Net's rater prompts. Its cost has only ever been counted in parameters, and parameter counts are known to predict latency on edge chips poorly.
This note proposes to measure, layer by layer, how much of each expert's style can be captured by a small low-rank weight correction (LoRA, a method that adds a trainable product of two thin matrices to a frozen weight). We hypothesize that inter-observer differences are low-rank and concentrated in the late decoder layers that shape boundaries. If that holds, a model can run the shared image encoder once and then produce all K expert hypotheses through a single stacked low-rank correction compiled as one static operation. That would make multi-expert segmentation feasible on ARM CPUs, embedded GPUs and edge accelerators. The hypothesis can be falsified: if a per-expert output layer does just as well, we will report that and recommend it instead.
Core Claim
The work makes two linked claims. The first is scientific. If one fine-tunes a full copy of a shared segmentation network for each expert and examines how each copy's weights differ from the shared weights, those differences have low effective rank. The rank grows toward the decoder's final stages. The encoder differences are negligible. The second claim is systems-level. Because rater-specific computation can then be confined to the last few layers, the K rater corrections can be concatenated into one rank-$Kr$ matrix, where $r$ is the per-rater rank. That matrix runs as a fixed-shape matrix multiply after a single encoder pass. Dynamic adapter switching tends to fail to compile on fixed-instruction accelerators and falls back to the host CPU; the stacked form avoids it.
The resulting model should be on or above the accuracy–latency frontier traced by per-rater heads, multi-decoder U-Nets and full-depth prompts or LoRA. Accuracy here means per-rater Dice and distribution-matching quality. Latency is measured on physical edge devices, not estimated from FLOPs.
Relation to Prior Work
Four methods motivate the problem and serve as our accuracy references. Pionono gives each rater a Gaussian in a latent space (Pionono). D-Persona learns a shared latent space and then attaches per-rater projection heads (D-Persona). MRNet reconstructs individual annotations with ConvLSTM modules (MRNet). The multi-decoder U-Net uses one decoder per annotator (Multi-decoder U-Net). All four are trained end to end and evaluated on desktop GPUs. MRNet's 29 FPS is the only speed figure among them. None asks where in the network rater differences actually reside.
The closest existing work is PU-Net. It inserts transformer blocks carrying rater prompts between the encoder and decoder of a frozen U-Net and trains about 0.1M parameters on RIGA. We differ in three ways. We use plain low-rank weight corrections rather than inserted attention blocks. We choose their depth and rank by measuring where rater differences live. We test on device, where PU-Net's inserted attention operators are exactly the kind of non-fused operation that can be slow.
Prior work on modality-specific LoRA for brain MRI attaches one adapter per scan sequence to a frozen diffusion backbone (Modality-LoRA). It supplies the mechanism of per-domain adapters on a shared base, but it models physics differences between scan types, not human disagreement, and it runs no edge tests.
Two edge-segmentation papers constrain the design. The analog in-memory computing study (Hamzaoui et al.) finds that U-Nets lose 0.15–0.22 accuracy under simulated analog noise while a Swin UNet loses 0.04. Its uncertainty comes from repeated Monte Carlo passes, which is the per-sample overhead we aim to remove. The ultrasound foundation-model work (Adapting VFMs) shows that adapters can run at about 77 FPS, but on a server GPU and against single-consensus labels.
Our insistence on physical measurement follows from LEDNet (LEDNet), which reports that desktop GPU latency does not predict ARM behavior. FDFlowNet (FDFlowNet) supports the same rule: operators without native accelerator support cause CPU fallback and large latency spikes. The multi-adapter serving systems Punica and CLIMB inspire the batched multi-adapter computation. They target dynamic, many-tenant language-model serving on server GPUs. Our setting has a small, fixed set of K raters, which is what lets us collapse the adapters into one static operator.
Research Program
Locating rater differences. We first locate where rater differences live. We train a shared backbone on all annotations. We test both a CNN U-Net and a Swin-style isotropic backbone, given the noise-robustness result above. We then fine-tune one full copy per rater. For each layer $\ell$ and rater $k$, we compute the difference $\Delta W_{\ell,k}$ between rater $k$'s weights and the shared weights. We record its effective rank, defined as the number of singular values needed to retain 90% of its squared Frobenius norm, together with its relative norm $\lVert \Delta W_{\ell,k} \rVert / \lVert W_\ell \rVert$.
We validate this analysis functionally, not just through spectra. We truncate each $\Delta W_{\ell,k}$ to rank $r$ and confine it to layers at depth $\ell \ge L_0$, then measure how much per-rater Dice survives. This produces a map over rank $r$ and start depth $L_0$. The per-rater fully fine-tuned models also give the "expert ceiling" against which unified models are judged.
Building and compiling the rater bank. For each rater $k$ we train $A_k$ (shape $r \times d$) and $B_k$ (shape $d \times r$) on layers from $L_0$ onward, with the shared weights $W_0$ frozen. We use PU-Net's mix-training idea of adding a consensus adapter, which also supports raters with missing annotations. At inference, the shared features $h$ pass once through $W_0$. All K corrections are then computed together:
$$Y = \mathbf{1}_K \otimes (W_0 h) + \mathrm{blockdiag}(B_1,\dots,B_K)\,[A_1; \dots; A_K]\, h,$$
Here $[A_1;\dots;A_K]$ stacks the down-projections into one $Kr \times d$ matrix, and $\mathrm{blockdiag}$ keeps each rater's up-projection separate. This yields K output maps from one static-shape computation.
Hardware testing. We export to ONNX and then to each device's native runtime: an ARM mobile CPU, an embedded GPU of the Jetson class, and an edge NPU/TPU. We log five quantities: end-to-end latency for all K hypotheses, peak activation memory, energy per scan, the number of operators that fall back to the CPU, and accuracy after 8-bit quantization of $W_0$. For the quantization condition, the adapters are kept in either 8-bit or 16-bit precision.
Baselines
The naive reference is a single U-Net trained on the soft average of all raters. It costs one pass. It cannot personalize, but its probability map can be thresholded for Q-Dice. Any proposal that loses to it on QUBIQ Q-Dice at higher latency has failed.
The strongest accuracy competitor on personalization is D-Persona. It reports a Generalized Energy Distance (GED, a score of how well the predicted set of masks matches the set of expert masks; lower is better) of 0.2359 on NPC-170, against 0.4465 for a Probabilistic U-Net. It also reports 76.19% mean per-rater Dice on NPC-170 and 89.17% on LIDC-IDRI. On QUBIQ, the strongest references are deep ensembles, with top-team average Dice between 0.80 and 0.85 (QUBIQ), and the multi-decoder U-Net, which reports 0.852, 0.871 and 0.870 on brain tumor tasks 1 and 2 and kidney, and 0.553 on prostate task 1. MRNet reports a 1.2% soft-Dice gain for the optic cup on RIGA at 29 FPS on desktop hardware. For PU-Net we found no reported Dice value on Lacuna, only its 0.1M trainable parameters, so we will rerun it.
All baselines will be retrained by us on identical splits with the same backbone where the method allows. Each receives the same hyperparameter-search budget in GPU-hours and is exported through the same toolchain. Their latency is measured on the same devices with the same batch size and input resolution.
The proposal beats a baseline if it meets two conditions. On accuracy, it must come within one standard deviation across seeds of D-Persona's GED and personalization Dice. On speed, it must run all K hypotheses at lower measured edge latency than the multi-decoder U-Net, PU-Net and the ensemble, with zero CPU-fallback operators.
Three ablations isolate the key ingredients. Full-depth LoRA tests whether late placement matters. Late LoRA without stacking, invoking K separate passes, tests the compiled bank. Random layer placement at matched parameter count tests whether the spectral analysis selects better layers than chance.
Why not just…?
Why not just put a separate output layer per rater on a shared network? This is the most serious threat. It is the extreme case of late placement, rank-unconstrained but one layer deep, and it would be nearly free on device. Our spectral analysis will settle it. If rater differences are concentrated in the final layer, per-rater heads will match late LoRA, and we will report heads as the recommended design. The contribution then becomes the measured evidence and the edge profiling. We expect boundary style to involve several decoder stages, because the multi-decoder U-Net needed whole decoders to place second in QUBIQ. That expectation remains a hypothesis to test, not a result we can claim.
Why not just predict one soft probability map? For QUBIQ's Q-Dice, which only thresholds a probability map, this may well suffice. That is why it is our naive baseline. A soft map cannot reproduce a specific rater's contour or a coherent set of plausible masks, and both GED and personalization Dice on NPC-170, LIDC-IDRI and RIGA measure exactly that. We will report the cases where the soft map wins.
Why not use Monte Carlo dropout or an ensemble? Both multiply full forward passes. The analog study (Hamzaoui et al.) relies on repeated sampling, and the QUBIQ organizers name ensemble cost as the deployment barrier. Monte Carlo dropout also captures model uncertainty rather than the systematic biases of individual raters, which is what Pionono's per-rater means encode. Our latency measurements will quantify the gap directly.
Why not use rater prompts as in PU-Net? Prompts may match our accuracy. The open question is whether the inserted attention blocks compile and fuse on edge accelerators. Our operator-fallback counts and latency measurements answer this directly.
Proposed Experiments
The first decisive check is the rank-and-depth map on RIGA (6 raters), NPC-170 (4 raters) and LIDC-IDRI (4 raters). The hypothesis survives if per-rater Dice from truncated, late-only corrections recovers most of the gap between the shared model and the per-rater full fine-tunes at small rank. It fails if the recoverable gap is spread across encoder layers or requires high rank.
The second check is the edge frontier. For each device, we plot GED and per-rater Dice against measured latency for all K hypotheses. The plot covers the soft-map U-Net, per-rater heads, the multi-decoder U-Net, PU-Net, D-Persona, a K-model ensemble, full-depth LoRA and the stacked late bank. Each point carries energy and fallback counts.
The third check is robustness. We quantize the shared weights to 8 bits while keeping adapters at higher precision, and test whether rater distinctions collapse; small low-rank corrections may fall below quantization noise. We also run a missing-rater condition in which each training scan keeps only a random subset of raters, a known weakness of current multi-rater methods.
The final check is QUBIQ Q-Dice across its prostate, kidney and brain tasks. This tells us whether the per-rater machinery adds anything over a single soft map on the challenge metric.
References
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- Diversified and personalized multi-rater medical image segmentation
- Learning Calibrated Medical Image Segmentation via Multi-Rater Agreement Modeling
- Uncertainty Quantification in Medical Image Segmentation with Multi-decoder U-Net
- Multi-rater Prompting for Ambiguous Medical Image Segmentation
- A Multi-Agent Diffusion Approach for MRI Anomaly Segmentation via Modality-Specific LoRA Specialization
- Analog In-Memory Computing with Uncertainty Quantification for Efficient Edge-based Medical Imaging Segmentation
- Adapting Vision Foundation Models for Real-Time Ultrasound Image Segmentation
- Lednet: A Lightweight Encoder-Decoder Network for Real-Time Semantic Segmentation
- Fdflownet: Fast Optical Flow Estimation Using A Deep Lightweight Network
- Punica: Multi-Tenant LoRA Serving
- CLIMB: Taming the LoRA Residency Cliff in Multi-LoRA Serving