In semantic segmentation, the standard approach is binary: a pixel is either part of an object or part of the background. While this works for isolated items, it fails catastrophically in dense environments like satellite imagery of cities or microscopic views of cells. When objects touch, a binary model sees a single continuous mass, a problem known as "instance merging" or "clumping."
To solve this, researchers have converged on a simple but powerful representational shift: treating boundaries not as an implicit edge between two classes, but as an explicit third class.
The Problem of Semantic Blurring
When a model like a U-Net is trained on a binary mask of buildings or nuclei, it learns to identify the "essence" of the object. However, at the exact point where two buildings touch, the visual features are often ambiguous. Without a specific penalty for merging, the network naturally "bleeds" the mask across the gap.
As noted by Rémi Delassus and Romain Giot in CNNs Fusion for Building Detection in Aerial Images for the Building Detection Challenge, binary classification fails to preserve the topological separation between neighboring entities. This is particularly problematic in high-density urban environments where the "gap" between structures might only be a few pixels wide.
The Three-Class Solution
The core intuition of this research direction is to transform the task into a three-class (ternary) problem:
- Interior (Core): The central body of the object.
- Boundary (Border): The thin interface where the object meets the background or another object.
- Background: Everything else.
By forcing the model to predict a dedicated "boundary" class, we create a mathematical "moat" around every instance. In Evaluation of Deep Learning Strategies for Nucleus Segmentation in Fluorescence Images, Juan C. Caicedo and colleagues demonstrated that this explicit representation reduces biologically critical errors—like merged cells—by up to 50%.
This "border-core" representation, as utilized in ParticleSeg3D, essentially erodes the object masks during training to create a guaranteed gap. Even if the model's predictions are slightly fuzzy, the presence of a predicted boundary prevents the "interior" pixels of two adjacent objects from ever touching.
Applications Across Domains
This mechanism has proven robust across vastly different scales and sensors:
- Agriculture: In the Fields of The World dataset, Hannah Kerner and Snehal Chaudhari use three-class masks to delineate contiguous agricultural plots. Without the boundary class, smallholder farms in Africa and Asia often merge into indistinguishable "super-fields."
- Medical Imaging: For spinal X-rays, Yao Chen uses instance-level boundaries in VertXNet to partition overlapping semantic masks of vertebrae. Similarly, Hongliang He in TopoSeg uses the three-class representation to enforce "boundary closure," ensuring that the predicted shapes are biologically plausible.
- Generative Modeling: Even in image synthesis, boundaries matter. Ting-Chun Wang and Jun-Yan Zhu showed in pix2pixHD that adding an instance boundary map as an input channel prevents GANs from merging adjacent cars or trees into a single distorted mass.
From Boundaries to Instances
Once a model predicts these three classes, how do we get individual objects? The boundary acts as a natural separator for post-processing algorithms.
A common technique is the Watershed Transform. As explored by Jiangqi Li and Minglei Li in their dual-path instance segmentation network, the predicted "interior" pixels serve as "seeds" or "markers," while the "boundary" pixels act as the ridges that prevent the seeds from flooding into one another.
Similarly, Vladimir Iglovikov and the authors of TernausNetV2 use an auxiliary "touching borders" channel. By subtracting these predicted borders from the primary semantic mask, they create distinct gaps that allow simple connected-component labeling to extract individual building footprints with high precision.
Why It Works: The Geometric Signal
The reason this approach is so effective is that it provides a stronger geometric signal than binary classification. A binary mask only tells the model where an object is. A three-class mask tells the model where an object ends.
By focusing the network's attention on the transition zones—the "unknown" regions in matting Yihan Hu or the "short-side edges" in text detection Chuhui Xue—we force the architecture to learn the high-frequency spatial details necessary for separation. This turns a difficult global problem (identifying every unique object) into a simpler local problem (identifying the edges between them).