Boundary Classes for Separating Touching Objects

This direction is split into other directions: boundary-aware instance separation. Its last version stays readable and citable here.

Version 1 of 163 papers82 conceptsfrozen 2026-10-04

In semantic segmentation, the standard approach is binary: a pixel is either part of an object or part of the background. While this works for isolated items, it fails catastrophically in dense environments like satellite imagery of cities or microscopic views of cells. When objects touch, a binary model sees a single continuous mass, a problem known as "instance merging" or "clumping."

To solve this, researchers have converged on a simple but powerful representational shift: treating boundaries not as an implicit edge between two classes, but as an explicit third class.

The Problem of Semantic Blurring

When a model like a U-Net is trained on a binary mask of buildings or nuclei, it learns to identify the "essence" of the object. However, at the exact point where two buildings touch, the visual features are often ambiguous. Without a specific penalty for merging, the network naturally "bleeds" the mask across the gap.

As noted by Rémi Delassus and Romain Giot in CNNs Fusion for Building Detection in Aerial Images for the Building Detection Challenge, binary classification fails to preserve the topological separation between neighboring entities. This is particularly problematic in high-density urban environments where the "gap" between structures might only be a few pixels wide.

The Three-Class Solution

The core intuition of this research direction is to transform the task into a three-class (ternary) problem:

  1. Interior (Core): The central body of the object.
  2. Boundary (Border): The thin interface where the object meets the background or another object.
  3. Background: Everything else.

By forcing the model to predict a dedicated "boundary" class, we create a mathematical "moat" around every instance. In Evaluation of Deep Learning Strategies for Nucleus Segmentation in Fluorescence Images, Juan C. Caicedo and colleagues demonstrated that this explicit representation reduces biologically critical errors—like merged cells—by up to 50%.

This "border-core" representation, as utilized in ParticleSeg3D, essentially erodes the object masks during training to create a guaranteed gap. Even if the model's predictions are slightly fuzzy, the presence of a predicted boundary prevents the "interior" pixels of two adjacent objects from ever touching.

Applications Across Domains

This mechanism has proven robust across vastly different scales and sensors:

  • Agriculture: In the Fields of The World dataset, Hannah Kerner and Snehal Chaudhari use three-class masks to delineate contiguous agricultural plots. Without the boundary class, smallholder farms in Africa and Asia often merge into indistinguishable "super-fields."
  • Medical Imaging: For spinal X-rays, Yao Chen uses instance-level boundaries in VertXNet to partition overlapping semantic masks of vertebrae. Similarly, Hongliang He in TopoSeg uses the three-class representation to enforce "boundary closure," ensuring that the predicted shapes are biologically plausible.
  • Generative Modeling: Even in image synthesis, boundaries matter. Ting-Chun Wang and Jun-Yan Zhu showed in pix2pixHD that adding an instance boundary map as an input channel prevents GANs from merging adjacent cars or trees into a single distorted mass.

From Boundaries to Instances

Once a model predicts these three classes, how do we get individual objects? The boundary acts as a natural separator for post-processing algorithms.

A common technique is the Watershed Transform. As explored by Jiangqi Li and Minglei Li in their dual-path instance segmentation network, the predicted "interior" pixels serve as "seeds" or "markers," while the "boundary" pixels act as the ridges that prevent the seeds from flooding into one another.

Similarly, Vladimir Iglovikov and the authors of TernausNetV2 use an auxiliary "touching borders" channel. By subtracting these predicted borders from the primary semantic mask, they create distinct gaps that allow simple connected-component labeling to extract individual building footprints with high precision.

Why It Works: The Geometric Signal

The reason this approach is so effective is that it provides a stronger geometric signal than binary classification. A binary mask only tells the model where an object is. A three-class mask tells the model where an object ends.

By focusing the network's attention on the transition zones—the "unknown" regions in matting Yihan Hu or the "short-side edges" in text detection Chuhui Xue—we force the architecture to learn the high-frequency spatial details necessary for separation. This turns a difficult global problem (identifying every unique object) into a simpler local problem (identifying the edges between them).

Papers in this version

Showing 60 of 63 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_boundary_classes_for_separating_touching_6b1ce28b,
  title = {Boundary Classes for Separating Touching Objects},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/boundary-classes-for-separating-touching-objects/rdr_6b1ce28b79334a748510ff1216a88354},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_boundary_classes_for_separating_touching_6b1ce28b_v1,
  title = {Boundary Classes for Separating Touching Objects},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/boundary-classes-for-separating-touching-objects/rdr_6b1ce28b79334a748510ff1216a88354/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}