When many objects of the same kind touch or overlap, as with cell nuclei in a tissue slide or leaves on a plant, a neural network's internal features near the contact line look the same for both objects. The network then tends to merge them into one blob. A separate line of work in image generation has shown that moving features around inside a network, called latent feature warping, works better than moving raw pixels. Feature warping resamples an intermediate feature map at displaced coordinates, and later layers can repair the result. We propose to use warping for separation rather than synthesis. Each pixel's features are warped by repeatedly sampling along a predicted path that stays inside its own object and leads to that object's core. After a few steps, the features on either side of a contact line have been pulled toward two different cores, so they stop resembling each other.
The opportunity exists because the two fields have so far met only in one way. Flow-based feature warping has been used to expand a single occluded object's features into its hidden region (Amodal Instance Segmentation via Prior-Guided Expansion). Nobody has asked what a warp does at the boundary between two touching instances. An unconstrained, smooth flow field should make that boundary worse, because it mixes neighbors' features. Our claim is that constraining the warp to follow the object's own geometry turns it into a tool for separation.
Core Claim
A feature warp whose sampling offsets are tied to a supervised, instance-internal direction field reduces under-segmentation of touching objects more than three alternatives: the same direction field used only for post-hoc pixel grouping, an unconstrained learned warp of equal capacity, and an embedding-orthogonality loss alone. We expect the gain to grow with the density of contacts in the image. We also predict a measurable mechanism. Feature similarity across contact boundaries should fall under the constrained warp and rise under an unconstrained one.
Research Program
The method builds on the center-vector idea from Accurate Nuclear Segmentation with Center Vector Encoding. That paper predicts, for every foreground pixel, a 2D displacement to its nucleus centroid, and it groups pixels by following those vectors. Vectors flip direction across a contact line, which reveals the boundary. We change two things.
The first change replaces straight-line centroid displacement with a unit direction field $u(p)$. This field is trained to match the gradient of the geodesic distance, measured inside the ground-truth instance mask, from pixel $p$ to the instance's medoid. Straight vectors leave the object for crescent or concave shapes, which is the same failure that limits star-convex polygon methods such as LSP-DETR. A geodesic field instead bends around concavities and never crosses into a neighbor. The idea echoes the intensity-gradient basins of 3D cell nuclei segmentation based on gradient flow tracking, but the field here is learned and supervised.
The second change uses the field to warp features at an intermediate decoder layer. This replaces using it only to group pixels at the end. Starting from decoder features $f_0$, we apply $K$ small steps:
$$f_{k+1}(p) = (1-g(p))\,f_k(p) + g(p)\,f_k\big(p + s\,u(p)\big),$$
where $s$ is a step length in feature-map pixels and $g(p)\in[0,1]$ is a learned gate. The gate lets the network leave already-unambiguous interior pixels alone, borrowing the soft-gating idea of Soft-Gated Warping-GAN. Sampling is bilinear and differentiable, so the instance heads train end to end through the warp. Because every sample is drawn from upstream along the pixel's own geodesic path, features stay inside one instance and converge toward a per-instance summary. In effect this is a few steps of differentiable mean-shift, restricted to object interiors.
The warped features then feed standard instance heads. In the main configuration these are Disco's coloring-plus-adjacency heads, so the warp is tested as an add-on to the strongest dense-cell method. Pixel grouping at inference still follows $u$, as in Li et al.
The generation literature motivates the design. It shows that warped intermediate features can be repaired by later layers rather than locking in interpolation errors (Deep Image Spatial Transformation for Person Image Generation; Chieh-Yun Chen et al.; Xin Yu et al.). The same literature constrains the design. Extreme warps corrupt latents (3D-Aware Face Editing via Warping-Guided Latent Direction Learning), and warps driven by wrong structure propagate errors (Soft-Gated Warping-GAN on low-resolution Market-1501). For these reasons we keep $s$ small, use few steps, and gate the update.
We deliberately depart from PGExp on three points. PGExp warps per isolated region of interest, conditions on shapes retrieved from a memory bank, and aims to expand an object into hidden area. Our warp operates densely over the whole image, needs no retrieval, and aims to separate neighbors. We do not attempt amodal completion of hidden nuclear area here. The dense-cell datasets in the file annotate only visible pixels, so such completion could not be evaluated honestly.
Baselines
The naive reference is the same encoder and decoder with Disco-style heads and no warp. Pixels are grouped by the geodesic field at inference, so the only difference from the full model is the warp module.
The strongest existing method on dense, conflict-heavy tissue is Disco. It casts instance separation as graph coloring with a third conflict color, plus a loss that pushes adjacent instances' embeddings toward orthogonality. It reports 50.87% Panoptic Quality (PQ) on GBC-FS 2025, 7.08 points above the next method. The file gives no per-dataset Disco numbers on PanNuke, DSB2018 or CryoNuSeg, only a 2.72% average improvement. For PanNuke, the reference is LSP-DETR's 48.2 multi-class PQ. On the Kumar et al. histology set, the reference is Li et al.'s 0.561 Aggregated Jaccard Index (AJI). We will rerun Disco ourselves on our splits with its published settings, rather than relying only on copied numbers. The proposal adds the warp to Disco, so beating it means the warped Disco exceeds our rerun of plain Disco. The bar is a PQ gain on GBC-FS that is larger than the seed-to-seed spread across three seeds, and in practice we target at least one PQ point.
Two ablations isolate the key ingredient. The first is an unconstrained warp of identical size, with offsets predicted freely by a small convolutional head as in deformable convolution or PGExp-style flow. It has no direction-field supervision and no tie to $u$. The second keeps the supervised field but feeds it to the heads as extra input channels without warping. All variants share the backbone, the training data, a fixed tuning budget of the same number of hyperparameter trials per variant, and the GPU-hour cap. Every inference-speed comparison is reported on the same GPU against Disco's reported 29.2 FPS on an RTX 4090.
Why not just…?
Why not just use deformable convolution or a freely learned flow warp? This is the real threat, because a free warp could in principle learn instance-respecting offsets on its own. Nothing in its training signal asks it to avoid sampling across contacts, though, and Gap 1 of the literature argues that smooth flows smear neighbors. The unconstrained-warp ablation settles the question. If it matches the constrained warp on GBC-FS and shows no rise in cross-boundary feature similarity, the geodesic constraint is unnecessary, and we would report that result.
Why not just group pixels with center vectors, as Li et al. already do? Grouping at the output fixes membership for pixels whose vectors are correct. It cannot repair the ambiguous features that cause wrong vectors at contacts in the first place. Li et al. also note straight-cut artifacts at touching interfaces. The no-warp baseline uses the identical field and grouping, so any gain is attributable to changing the features.
Why not just rely on Disco's adjacency-orthogonality loss? That loss already pushes neighbors apart in embedding space. However, it acts only on per-instance embeddings and only through the training objective. Our warp changes the features that the embeddings are computed from. We test the two together and separately. If warp-without-loss and loss-without-warp are each weaker than both combined, they are complementary. If the loss alone matches, the warp adds nothing.
Why not just use a larger backbone or diffusion shape priors? AISDiff relies on text-conditioned diffusion attention. It is slow, and its prompts carry little meaning for nuclei. Model size is controlled by matching the parameter count of the warp module with extra decoder width in the no-warp baseline.
Proposed Experiments
We train and evaluate on GBC-FS 2025, which has more than 30% topological conflicts, and on PanNuke, which Disco reports as fully bipartite. We use PQ and multi-class PQ, along with AJI on the Kumar et al. set. CVPPP A1 serves as a non-star-convex, heavily overlapping case, measured by count difference against RIS's 0.2 from Recurrent Instance Segmentation and by mask quality.
The first decisive check is density. We stratify test tiles by contacts per instance. The prediction is that the warp's gain over plain Disco increases with contact density and is near zero on isolated nuclei. A flat gain would suggest generic extra capacity instead.
The second check is mechanistic. We measure the mean cosine similarity of decoder features in thin bands on either side of ground-truth contact lines, compared with within-instance similarity. The constrained warp should lower the cross-boundary ratio. The unconstrained warp should raise it. If both move the same way, the stated mechanism is wrong even if the accuracy numbers look good.
The third check targets shape. On nuclei whose ground truth is non-star-convex, and on CVPPP leaves, the geodesic field should beat straight centroid vectors used inside the same warp. This would show that the field's geometry, not warping alone, carries the effect.
Finally, we report inference speed and whole-slide throughput on TCGA alongside LSP-DETR's 0.46 s/mm², because a separation gain that costs gigapixel-scale speed would limit practical use.
References
- Amodal Instance Segmentation via Prior-Guided Expansion
- Accurate Nuclear Segmentation with Center Vector Encoding
- LSP-DETR: Efficient and Scalable Nuclei Segmentation in Whole Slide Images
- 3D cell nuclei segmentation based on gradient flow tracking
- Soft-Gated Warping-GAN for Pose-Guided Person Image Synthesis
- Deep Image Spatial Transformation for Person Image Generation
- FashionMirror: Co-attention Feature-remapping Virtual Try-on with Sequential Template Poses
- Face Hallucination with Tiny Unaligned Images by Transformative Discriminative Neural Networks
- 3D-Aware Face Editing via Warping-Guided Latent Direction Learning
- Disco: Densely-overlapping Cell Instance Segmentation via Adjacency-aware Collaborative Coloring
- Amodal Instance Segmentation with Diffusion Shape Prior Estimation
- Recurrent Instance Segmentation