Training a medical image segmenter normally requires an expert to trace every organ boundary pixel by pixel, which is slow and expensive. A cheaper option is a scribble: one quick stroke drawn through each structure. The hard part is turning a few annotated strokes into a full mask. Current methods do this outside the network with slow classical tools such as random walks, conditional random fields, or tree-based graph propagation, or they rely on hand-set distance rules. This note proposes doing the spreading inside the network. Linear attention is a variant of transformer attention whose cost grows linearly, not quadratically, with image size. We would reuse a linear-attention layer as the operator that carries scribble labels to every unlabeled pixel. Concretely, the label maps go in where the attention values normally sit, so each pixel receives a feature-similarity-weighted vote from the annotated pixels at linear cost.
The obstacle is a known weakness of linear attention. Its weights are all non-negative, so every pixel receives some vote from every scribble, and the resulting label maps come out blurred. Gated differential linear attention fixes this by subtracting two attention maps, which cancels the shared background signal. In our setting it should therefore give sharp, boundary-respecting soft labels. The operator is trained by a simple self-check: hide some scribble pixels and ask it to recover their labels from the rest. If this works, scribble-supervised training no longer needs an external refinement stage. Because the cost is linear, the same operator can also spread labels through whole 3D volumes, where graph and conditional-random-field refiners become expensive.
Core Claim
A differential linear-attention operator, trained only by predicting held-out scribble pixels, gives pseudo-labels at least as accurate as distance-decayed soft labels and tree-based affinity propagation, at lower cost. The subtraction step is what makes it work: an otherwise identical non-differential operator should produce visibly more label bleeding across thin structures such as the myocardium.
Relation to Prior Work
The attention design comes from Gated Differential Linear Attention. That paper builds a linear-time decoder that subtracts two kernelized attention paths with a learned per-channel scale and adds head gating. It reaches 85.32 Dice on Synapse CT at 6.85G FLOPs, but it is trained only with dense masks. We use its operator for a different job: carrying labels, not decoding features.
Label-efficient Segmentation via Affinity Propagation (APro) is the closest competitor. It spreads soft labels over a minimum spanning tree in $O(N \log N)$ time and reports 76.6% mIoU with scribbles on Pascal VOC. Its affinities, however, come from a fixed image graph that is never trained to predict labels, and it has only been tested on 2D natural images. Our affinities are learned from the scribbles themselves.
MaCo motivates the soft-label approach and sets the bar. Its continuous pseudo-labels decay with distance from the scribble and give 8.3–23.3% Dice gains over hard pseudo-labels, and the full method reaches 90.4% Dice on ACDC. Those labels depend only on geometry. Ours depend on learned appearance.
ScribFormer shows that global attention helps fill out whole organs from scribbles (0.888 Dice on ACDC), using quadratic attention on downsampled features. It uses attention to shape features, not to move labels.
Two papers constrain the design. Can et al. show that iterative conditional-random-field relabeling with dropout-based uncertainty filtering works, coming within 2.9% Dice of full supervision on ACDC, but each round requires retraining. Agarwal et al. report that adding a dense conditional random field sharply cuts recall. Any propagator we build must therefore be checked for recall, not only precision.
Research Program
The operator works as follows. Let $Y_s \in \mathbb{R}^{N \times C}$ be the one-hot scribble labels over $N$ feature-map tokens and $C$ classes, with zero rows for unlabeled tokens, and let $m \in \{0,1\}^N$ mark which tokens are labeled. Let $\phi$ be a non-negative kernel feature map, and let $Q_i, K_i$ be query and key projections of the decoder features in two subspaces $i=1,2$. The propagated labels are
$$\hat{Y} = \mathrm{norm}\Big(\phi(Q_1)\big(\phi(K_1)^\top Y_s\big) - \lambda \odot \phi(Q_2)\big(\phi(K_2)^\top Y_s\big)\Big),$$
where $\lambda$ is the learned per-channel scale and "norm" does two things: it clips negative values, and it divides each row by the same expression computed with $m$ in place of $Y_s$, which measures how much scribble mass reached that token. Both terms reorder the matrix product as $\phi(Q)(\phi(K)^\top Y_s)$, so the cost stays $O(NC)$.
Training has two parts. First, the operator learns by self-checking. At each step a random 30–50% of scribble tokens are withheld, and $\hat{Y}$ computed from the remaining tokens is scored with cross-entropy on the withheld ones. This gives the operator direct supervision without any dense mask. Second, the segmentation head trains on its usual partial cross-entropy on scribbles, plus a confidence-weighted loss toward $\hat{Y}$ on unlabeled tokens.
The propagator's queries and keys are taken from the decoder's own attention layers. This keeps added parameters small. It also means the decoder must learn features that pull same-structure pixels together, which addresses the open question of whether gated differential attention stays stable under sparse gradients. We will track $\lambda$, the gate activations, and how different the heads remain from each other throughout training to detect collapse.
The 3D extension needs no new machinery. On ACDC and MSCMRseg, we compute $\phi(K)^\top Y_s$ once over all tokens of all slices in a volume, so scribbles on one slice can inform the adjacent slices. Peak memory and wall-clock time will be reported against a 3D minimum-spanning-tree version of APro.
Baselines
The naive reference is a PVT-GDLA network trained with partial cross-entropy on scribbles only, using the same encoder and decoder as our method but without propagation. It measures what the architecture achieves on its own. Our method must improve on it in every structure, and especially in the myocardium.
The strongest published competitor is MaCo, which reports 90.4% average Dice on ACDC, 88.5% on MSCMRseg, and 80.5% on NCI-ISBI (MaCo). ScribFormer's 0.888 on ACDC is a second published reference (ScribFormer). Because backbone choice alone could move Dice by a point or more, we will also reimplement MaCo's continuous pseudo-labels on our PVT-GDLA backbone. That gives an equal-architecture comparison with the same scribbles, data splits, epochs, and hyperparameter search budget, for example 20 trials per method. Beating MaCo means exceeding 90.4% ACDC Dice on the published comparison and also winning the equal-backbone comparison on at least two of the three datasets.
APro is the strongest competitor for the specific question of how to spread labels. No reported result was found on Lacuna for APro on any medical dataset, so we will run it ourselves. We will plug it into the same backbone in place of our operator, at matched training cost, and run it in both 2D and 3D form. Beating APro means higher pseudo-label Dice on the training set at equal or lower propagation time.
Two ablations isolate the key ingredients. The first sets $\lambda=0$ and removes the gate, leaving plain linear-attention propagation; this tests the claim that the subtraction is what makes the method work. The second removes the held-out-scribble loss, so the affinities are learned only indirectly through the segmentation loss. A third reference is softmax attention used as the propagator at reduced resolution. It shows how much accuracy, if any, the linear form gives up relative to quadratic attention.
For context, fully supervised ACDC reaches 91.2% Dice in Can et al. and 0.891 in the multi-constraint self-training paper.
Why not just…?
Why not just use MaCo's distance-decayed pseudo-labels? This is a serious alternative: they are free to compute and already strong. Their weakness is that they ignore image content. A pixel just outside the myocardium but close to a myocardium scribble gets high confidence for the wrong class. The equal-backbone comparison settles this question. We will also measure pseudo-label error within a few pixels of true boundaries, where the two approaches should differ most. If learned propagation does not beat distance decay there, the central claim fails.
Why not just bolt APro onto an existing scribble model? APro already provides fast, topology-aware propagation and could plausibly match us in 2D. We expect learned affinities to matter on low-contrast MRI, where the image graph APro builds its tree from may contain shortcuts across boundaries, a risk APro's own analysis notes. We also expect a cost advantage in 3D. The direct swap described in the Baselines section decides this. If APro matches us in 2D, we will report that honestly and the contribution narrows to the 3D and learned-affinity cases.
Why not just train PVT-GDLA with partial cross-entropy and let its attention fill in shapes implicitly? ScribFormer suggests that global attention alone helps substantially. The naive baseline measures exactly this. The non-differential ablation then tells us whether any gain comes from explicit propagation rather than from the architecture.
Why not just use a conditional random field? Agarwal et al. found that it cuts recall, and Can et al. needed repeated retraining rounds around it. We will include a dense conditional random field on the naive baseline's outputs and report both precision and recall.
Decisive Checks
The experiments use ACDC, MSCMRseg, and NCI-ISBI with their standard scribble annotations, reporting per-structure Dice and 95th-percentile Hausdorff distance. The robustness protocol follows MaCo's: drop 50% of scribble pixels, where MaCo loses 3.3% Dice. We also test extreme sparsity using 5% scribbled training data, following the adversarial attention gate work.
Three outcomes would be decisive. First, the operator's pseudo-label Dice on training data should exceed both distance-decay labels and APro. Second, the non-differential ablation should lose most of that gain, concentrated on thin structures. Third, 3D propagation should improve Dice on slices far from any scribble while using less memory than 3D APro.
As a stretch test of the 3D claim, we will apply the method to DeepLesion, propagating RECIST tumor-diameter lines through each CT volume. There we compare against the 92.6% Dice reported by AHRNet with level-set loss on the 2D test masks.
References
- Gated Differential Linear Attention: A Linear-Time Decoder for High-Fidelity Medical Segmentation
- Label-efficient Segmentation via Affinity Propagation
- From Few to More: Scribble-Based Medical Image Segmentation via Masked Context Modeling and Continuous Pseudo Labels
- ScribFormer: Transformer Makes CNN Work Better for Scribble-based Medical Image Segmentation
- Learning to Segment Medical Images with Scribble-Supervision Alone
- Weakly Supervised Lesion Co-Segmentation on Ct Scans
- A Semi-Supervised Framework for Cardiac MRI Segmentation via Multi-Constraint Collaborative Self-Training
- Learning to Segment From Scribbles Using Multi-Scale Adversarial Attention Gates
- Weakly-Supervised Universal Lesion Segmentation with Regional Level Set Loss