When two translucent cells overlap on a cytology slide, the pixels in the shared region belong to both cells at once. The evidence that says which two cells they are, namely the nuclei, can sit far from that region, often well outside what a convolutional network sees from one spot. We propose a model in which every pixel predicts a small set of arrows, one for each cell it belongs to, each pointing at that cell's nucleus. The arrows are computed by an encoder that interleaves global attention with local convolution: attention lets a pixel find its distant nucleus, and convolution keeps the arrows sharp where cells touch. Convolution here means sliding a small filter over the image. Attention means letting every location directly read from every other location.
The opportunity is partly a method and partly a test. Methods built on center vectors, which assign each pixel a two-dimensional arrow to its object's center, separate touching nuclei well. They assume one owner per pixel and a target that is close by. Both assumptions break for overlapping cytoplasm. We claim that the receptive field a center-vector model needs grows with object radius rather than boundary width. If that is right, global routing should matter for large overlapping cells and hardly at all for small nuclei. The experiments are built to check that prediction, not only to post a higher score.
Core Claim
Two things are new. First, the center-vector representation is extended from one arrow per pixel to a set of up to $K$ arrows per pixel, so that overlap regions are represented directly instead of being decomposed and recombined. Second, we claim a specific division of labor in a sequential local-global block. The attention step fixes errors whose cause is distance: the pixel cannot see its target. The convolution step fixes errors whose cause is contact: the arrow must flip direction across a thin boundary. This gives a falsifiable signature. For purely local models, arrow error should rise sharply once the distance from a pixel to its nucleus exceeds the model's effective receptive field. For the hybrid model, error should stay flat. On small-object benchmarks the two should be nearly indistinguishable.
Prior Work and Where This Sits
The representation comes from Accurate Nuclear Segmentation with Center Vector Encoding. That paper trains a convolutional network to predict one offset to the nuclear centroid. At touching borders the offsets flip by 180 degrees, which separates instances, and it reports an Aggregated Jaccard Index of 0.561 on the Kumar benchmark. We reuse this idea but make it set-valued, and we question its purely convolutional backbone.
The overlap regime comes from DoNet. DoNet splits clusters of translucent cervical cells into intersection and complement regions and recombines them inside a Mask R-CNN pipeline, reaching 64.02 mAP on ISBI2014 and 49.43 on CPS. Our multi-valued field represents the same intersection region without region proposals or a separate recombination step. DoNet is our strongest competitor.
Two papers bound the representation space from opposite sides. LSP-DETR predicts star-convex polygons with linear tile attention. It is fast (0.46 s/mm² on whole slides), but polygons cannot describe the non-convex cytoplasm sheets in which overlaps occur. Disco handles dense touching nuclei through graph coloring. It addresses adjacency but not shared pixels.
The architectural question is framed by a disagreement. LKCell argues that 13×13 depthwise kernels give a sufficient "large-but-local" field, reaching 0.5080 mPQ on PanNuke at a fraction of a foundation model's FLOPs. Hybrid designs argue that global routing is needed. These include the dual-branch visual Conformer, which couples a ResNet branch and a DeiT branch through feature coupling units, and ConvFormer, which places depthwise convolutions inside the Transformer's feed-forward layers and trains from scratch to 0.925 Dice on MM-WHS. Neither hybrid was tested on instance separation.
CCNet supplies the global operator we use. Its criss-cross attention lets each pixel attend along its row and column. Applied twice, this reaches every pixel at $O(N\sqrt{N})$ cost with 11× less memory than dense non-local blocks, which keeps the attention affordable at the resolutions where cell boundaries live.
Finally, Is PEFT Enough for Cell Segmentation? constrains the design. It shows that adapting Cellpose-SAM on LIVECell plateaus near 0.56 AP however it is fine-tuned. That is a reason to train a structure-aware model rather than adapt a frozen one.
Research Program
The encoder is a stride-4 stem followed by sequential blocks in the speech Conformer order: feed-forward, two passes of criss-cross attention, a depthwise convolution module, then feed-forward. This follows the block used in non-autoregressive speech recognition. We choose the sequential form over the dual-branch visual Conformer because it avoids the cross-branch alignment overhead that the research file flags as an open cost. Whether that choice loses accuracy is tested as an ablation.
A light decoder upsamples to full resolution and produces three outputs:
- a nucleus-center heatmap;
- $K$ slots per pixel, each holding an occupancy probability and a two-dimensional offset;
- a foreground mask.
We set $K=3$ and will check the share of triple-overlap pixels in ISBI2014 and CPS.
Training uses full (amodal) overlapping instance masks, which both cytology datasets provide. For each pixel $p$ with ground-truth owner set $S_p$, the loss is a per-pixel matching between slots and owners:
$$ \mathcal{L}_p=\min_{\sigma}\sum_{k=1}^{K}\Big[\mathrm{BCE}\big(o_{p,k},\,\mathbb{1}[\sigma(k)\in S_p]\big)+\mathbb{1}[\sigma(k)\in S_p]\,\big\|v_{p,k}-(c_{\sigma(k)}-p)\big\|_1\Big] $$
Here $o_{p,k}$ is slot $k$'s occupancy probability, $v_{p,k}$ is its predicted offset, $c_i$ is the nucleus center of instance $i$, and $\sigma$ ranges over assignments of slots to owners and empty. With $K\le 3$ the minimum can be computed by brute force.
Inference runs on the GPU:
- Detect nucleus peaks from the heatmap.
- Let each active slot cast a vote at $p+v_{p,k}$.
- Assign the slot to the nearest peak within a radius.
- Each instance mask is the set of pixels with any slot assigned to that peak. Masks may overlap by construction.
No watershed is used. This removes the CPU-bound post-processing that both LKCell and HoVer-style pipelines depend on.
The analysis instrument is error stratified by distance. For every foreground pixel we record its distance $d$ to its true nucleus and the angular error of its matched arrow. We also record whether it lies within three pixels of a contact boundary. We measure each model's effective receptive field empirically, as the radius containing most of the input-gradient mass of a center output unit, and plot error against $d$ divided by that radius. This plot, not the headline metric, is what supports or refutes the core claim.
Baselines
The naive reference point is the single-arrow center-vector model of Center Vector Encoding on a plain ResNet-style encoder at matched parameters. It represents an overlap pixel by only one owner. Beating it shows the value of the representation alone. To isolate the representation from the encoder, we also run our full encoder with $K=1$.
The strongest existing method on the overlap question is DoNet, which reports 64.02 mAP on ISBI2014 and 49.43 mAP on CPS. It is the strongest competitor because it was designed for exactly these translucent overlaps. We will report its published numbers. We will also retrain it under our splits, augmentation, and epoch budget, using the authors' code if available and otherwise their published description, and give it the same number of hyperparameter trials (twenty) that we give ourselves. Beating it means exceeding its retrained mAP on both datasets with non-overlapping three-seed confidence intervals, and matching or exceeding its published figure.
The strongest receptive-field competitor is an LKCell-style encoder: our network with every attention step replaced by reparameterized 13×13 depthwise convolutions, tuned to match FLOPs within 10%, with the same multi-valued head. On nuclei its reported figure is 0.5080 mPQ on PanNuke, and we will rerun it on PanNuke under our protocol. Because this variant shares our head, it is also the decisive ablation for the attention ingredient. A second ablation removes the convolution module and keeps attention only, which tests the low-pass concern at contact zones. A third swaps our sequential block for a dual-branch block with feature coupling units, following the visual Conformer.
For LIVECell, the reference is the plateau reported in Is PEFT Enough: full fine-tuning at 0.5627 AP_COCO and base Cellpose-SAM at 0.5447. For the Kumar dataset, the reference is the 0.561 AJI from center vector encoding. For star-convex comparison on PanNuke, we cite LSP-DETR's reported 48.2 mPQ.
Why not just…?
Why not just use large kernels, as LKCell does? This is the main threat, and on nuclei we expect it to win or tie. A 13×13 kernel at stride 4 already spans a typical nucleus and its neighbors. Our claim is narrower: the large-kernel variant should fall behind only where the pixel-to-nucleus distance exceeds its effective receptive field, which happens in cytoplasm far from the nucleus. The FLOP-matched ablation settles this. If it matches the hybrid on ISBI2014 and CPS and shows the same flat error-versus-distance curve, the attention claim is wrong. The contribution would then shrink to the multi-valued representation, and we would report it that way.
Why not just append criss-cross attention to the end of a convolutional backbone? This is cheaper and gives a global field. It is a real alternative. ConvFormer's ablation, in which removing its transformer module cost 9.6% F1 on ultrasound, suggests that integrating attention throughout the network beats bolting it on, but that evidence comes from semantic segmentation. We will therefore run "CNN plus one CCNet head at the deepest stage" at matched compute. If it matches interleaved blocks, we will adopt it, since it is simpler.
Why not just keep DoNet's decompose-and-recombine pipeline with a stronger backbone? That would test whether the gain comes from the encoder rather than the representation. We will swap our encoder into DoNet. If DoNet with our encoder matches our full model, the multi-valued field adds nothing beyond simplicity and speed, and we will say so. The representation must earn its place through mAP at overlap pixels specifically, or through removing the proposal stage at equal accuracy.
Why not just fine-tune a segmentation foundation model? On LIVECell, Is PEFT Enough found adapters, loss changes, and full fine-tuning all stuck near 0.56 AP. That is direct evidence that adaptation alone does not resolve dense separation. We include LIVECell to see whether a model trained from scratch with an explicit separation target clears that band.
Proposed Experiments and Decisive Checks
The primary evaluation is mAP on ISBI2014 and CPS, plus mAP computed only on instances that overlap another instance. Secondary evaluations are mPQ on PanNuke, AJI on Kumar, and AP_COCO on LIVECell. All models get three seeds, identical data and augmentation, matched FLOPs within 10%, and twenty tuning trials each. We also report throughput, to check that GPU vote grouping is faster than watershed post-processing.
Three outcomes would decide the claim:
- Contrast between object scales. On cytoplasm, the hybrid beats the large-kernel variant by a clear margin. On PanNuke nuclei, the gap is within seed noise. If the hybrid wins equally on both, the effect is extra capacity rather than reach. If it wins on neither, attention is unnecessary.
- Shape of the error curve. For local models, error rises past distance divided by receptive-field radius ≈ 1, and the hybrid's curve stays flat. This is the mechanistic evidence.
- Contact-zone effect of convolution. The attention-only ablation degrades arrow accuracy specifically within three pixels of contact boundaries, while far-pixel accuracy is unchanged. This would support the division of labor and address the concern that global attention over-smooths edges.
If all three hold and the full model beats retrained DoNet on overlap-only mAP, the proposal has shown both a better representation for shared pixels and a principled account of when global context helps instance separation.
References
- Accurate Nuclear Segmentation with Center Vector Encoding
- DoNet: Deep De-overlapping Network for Cytology Instance Segmentation
- LSP-DETR: Efficient and Scalable Nuclei Segmentation in Whole Slide Images
- Disco: Densely-overlapping Cell Instance Segmentation via Adjacency-aware Collaborative Coloring
- LKCell: Efficient Cell Nuclei Instance Segmentation with Large Convolution Kernels
- Conformer: Local Features Coupling Global Representations for Visual Recognition
- ConvFormer: Combining CNN and Transformer for Medical Image Segmentation
- CCNet: Criss-Cross Attention for Semantic Segmentation
- Is PEFT Enough for Cell Segmentation? An Empirical No-Go Result on Frozen Foundation Models
- Pushing the Limits of Non-Autoregressive Speech Recognition