Centroid Relative Displacement Fields for Instance Segmentation

20 concepts13 papers1 proposal

In computer vision, separating individual object instances remains a distinct challenge from semantic classification. While semantic segmentation excels at labeling what class a pixel belongs to, it struggles when multiple objects of the same class touch or overlap. In such cases, standard pixel-connectivity approaches merge distinct entities into a single amorphous blob.

Traditional workarounds, such as bounding-box-based proposals, struggle with highly non-convex, interwoven, or elongated shapes (like text lines, chromosomes, or neural filaments). To solve this, a powerful geometric paradigm has emerged: centroid-relative displacement fields. Instead of predicting discrete boundaries or absolute distances, models learn to predict a continuous vector field where every foreground pixel points directly toward the geometric center of its parent instance.

The Intuition: Pixels with a Compass

To understand why this approach works, imagine every pixel inside an object is equipped with a compass pointing toward that object's center of mass.

If two objects of the identical semantic class are side-by-side, a standard semantic segmentation network sees a uniform wall of identical class labels. However, if the network is trained to predict the relative direction or offset from each pixel to its respective instance centroid, the geometric landscape changes entirely. Pixels on the left side of the boundary point rightward toward their object's center, while pixels just across the boundary on the right point leftward toward their object's center.

This creates a sharp phase transition—a sudden "flip" in the predicted vector directions—right at the contact boundary. By transforming instance segmentation into a geometric coordinate mapping task, the network creates natural topological boundaries and "sinks" that make instance separation straightforward.

Stability Over Distance Transforms

A naive alternative to vector fields is predicting an absolute distance transform (i.e., how many pixels away the nearest boundary or center is). However, absolute distances scale poorly with variations in object size and shape.

As shown by Bai et al. in their work on the Deep Watershed Transform for Instance Segmentation, predicting a unit vector field pointing toward the center of a parent object serves as a much more stable intermediate training signal than absolute distances. The direction vector is scale-invariant; whether an object is massive or tiny, the directional cue remains a consistent, local geometric property that simplifies how the network learns diverse object morphologies.

This representation is highly flexible. For instance, Zhang et al. demonstrated in Synthetic instance segmentation from semantic image segmentation masks that predicting these displacement fields allows networks to successfully extract instance-level information even when starting from class-agnostic semantic masks.

Resolving Overlaps and Dense Crowds

The power of centroid-relative mapping becomes evident in dense or highly overlapping environments where bounding boxes fail.

In biological imaging, for example, cells and nuclei frequently pack tightly together. The authors of CDNet: Centripetal Direction Network for Nuclear Instance Segmentation leveraged this by training a network to output a "centripetal direction" for every pixel. By calculating the directional differences between neighboring pixels, their model precisely isolates touching nuclei. Similarly, Ren et al. showed in End-to-End Instance Segmentation with Recurrent Attention that supervising the model with relative angles from pixels to centroids provides a rich representational signal that acts as a powerful boundary cue in high-density scenes.

When objects explicitly overlap and cross through one another—such as chromosomes in a clinical karyotype image—pixels at the intersection become ambiguous. Kluvanec et al. addressed this in Using Orientation to Distinguish Overlapping Chromosomes by using orientation vectors as a unique geometric signature to untangle overlapping structures and logically assign intersection pixels to their true parent instances.

[ Instance A Center ] <--- (Pixel Vector Field) --- [ Boundary ] --- (Pixel Vector Field) ---> [ Instance B Center ]
                                                   ^
                                        Directional "Flip" occurs here

From Vectors to Masks: The Assignment Policy

Once a network successfully predicts these displacement fields, how are they converted back into discrete instance masks? The predicted vectors essentially act as a voting mechanism or a coordinate routing system.

  1. Proximity-Based Assignment: In Single-Shot Panoptic Segmentation, Weber et al. utilize a dedicated network branch to predict instance centers, and then employ a geometric assignment policy that attributes pixels in overlapping regions to the nearest predicted center.
  2. Multi-Task Geometric Regression: For complex, arbitrarily shaped objects like curved text lines, Wang et al. demonstrate in A Single-Shot Arbitrarily-Shaped Text Detector based on Context Attended Multi-Task Learning that combining center offsets with vertex/corner offsets prevents adjacent text instances from merging into one another.
  3. Spatiotemporal Grouping: In video domains, Ci et al. show in Video Object Segmentation by Learning Location-Sensitive Embeddings that predicting spatial offsets to an object's centroid allows a model to group pixels into coherent instances across frames, ignoring background elements that might share a similar visual appearance.

Generalization Across Dimensions and Supervision Levels

Because centroid-relative mapping relies on fundamental spatial geometry, it generalizes remarkably well across different dimensions and supervision constraints.

Weakly Supervised Learning

Labeling pixel-perfect masks is incredibly expensive. To bypass this, Ahn et al. developed IRNet, showing that displacement fields can be learned from weak, image-level class labels to partition global class attention maps into discrete, localized masks. This exact geometric mechanism was later utilized by Gupta et al. in Deep Learning for Morphological Identification of Extended Radio Galaxies using Weak Labels to successfully isolate multi-component radio galaxies in crowded astronomical surveys without requiring manual pixel masks.

3D and Filamentous Structures

The concept translates naturally from 2D pixel grids to 3D voxel spaces. Bernard Ghanem demonstrated in 3D Instance Segmentation via Multi-Task Metric Learning that forcing each spatial voxel to predict a unit vector pointing toward its parent object's center of mass provides a highly stable reference for separating touching objects in complex indoor environments.

Even in the most extreme topological cases—such as tracking long, thin, interwoven neural filaments in the brain—the authors of FISBe: A Real-World Benchmark Dataset for Instance Segmentation of Long-Range thin Filamentous Structures highlight local offset prediction as a core strategy. By estimating the spatial displacement between pixels and instance centers, models can maintain directional consistency and differentiate between overlapping structures of identical intensity.

Key Foundational Work

MaskLab: Instance Segmentation by Refining Object Detection With Semantic and Direction Features (2018) — contributed 2 concept elements to this direction.
End-to-End Instance Segmentation with Recurrent Attention (2017) — contributed 2 concept elements to this direction.
Deep Watershed Transform for Instance Segmentation (2017) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_centroid_relative_angle_mapping_8e488466,
  title = {centroid-relative angle mapping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/centroid-relative-angle-mapping/rdr_8e488466baa74ad3bfe4b915e5842997},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_centroid_relative_angle_mapping_8e488466_v1,
  title = {centroid-relative angle mapping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/centroid-relative-angle-mapping/rdr_8e488466baa74ad3bfe4b915e5842997/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake