Disentangling Dense and Overlapping Scene Text Instances

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2129 papers161 conceptsfrozen 2026-10-04

In the evolution of computer vision, scene text detection has shifted from identifying individual characters to a more holistic approach: semantic segmentation. Instead of searching for local candidates, modern models like those proposed by Cong Yao and Xiang Bai in Scene Text Detection via Holistic, Multi-Channel Prediction treat text detection as a pixel-wise labeling task.

However, this shift introduced a fundamental challenge known as text adhesion. When two words are positioned in extreme proximity, a standard segmentation network often predicts a single, continuous blob of "text" pixels. Without a clear boundary, the system cannot tell where one word ends and the next begins.

The Geometry of Adhesion

The core problem lies in the loss of instance-level identity. In a dense scene—like a menu, a cluttered storefront, or a historical document—text lines often touch or overlap. Standard Fully Convolutional Networks (FCNs) excel at finding "text-ness," but they struggle to resolve the narrow valleys of background pixels that separate distinct entities.

As noted by Xuejian Rong and Yingli Tian in Unambiguous Text Localization and Retrieval for Cluttered Scenes, treating text as a generic object class ignores its unique spatial distribution. When text follows a curved path, as explored by Lianwen Jin in Detecting Curve Text in the Wild, traditional quadrilateral bounding boxes overlap even more aggressively, creating further ambiguity for detection algorithms.

The Shrink-Mask Solution

To combat adhesion, researchers introduced the shrink-mask strategy. Instead of training a model to predict the full extent of a text instance, they train it to predict a "shrunk" version of the text—essentially the skeleton or the central kernel of the word.

By supervising the network with these smaller kernels, the model naturally leaves "gaps" between adjacent words. Minghui Liao and colleagues popularized this with Real-Time Scene Text Detection with Differentiable Binarization. Their Differentiable Binarization (DB) module allows the network to adaptively set local thresholds for every pixel, making the separation of instances part of the end-to-end learning process rather than a post-processing heuristic.

Beyond Simple Kernels: Nested and Complex Layouts

While shrink-masks solve basic adhesion, they fail in "nested" scenarios—where a small word is located inside the spatial boundaries of a larger character, or when text is so thin that the "shrunk" kernel disappears entirely.

Researchers have developed several sophisticated mechanisms to handle these edge cases:

  1. Adaptive Offsets: Chuang Yang proposed Adaptive Shrink-Mask for Text Detection, which uses learned offsets to decouple text contours from rigid masks. This allows the model to recover the full shape of the text even when the initial prediction is just a thin central line.
  2. Border Semantics: Instead of just looking at the center, Chuhui Xue and Shijian Lu introduced Accurate Scene Text Detection through Border Semantics Awareness. By explicitly classifying four different types of borders, the model gains a structural understanding of where one instance must end.
  3. Mountain Climbing: In TextMountain, Jun Du reimagines text instances as "mountains." The center of a word is the peak, and the border is the foot. By predicting a direction vector field, pixels "climb" toward their respective peaks, naturally grouping themselves into distinct instances even in dense clusters.

Relational and Contextual Refinement

The most recent frontier involves moving beyond pure geometry to relational context. Models like the Text Relation Network (TRN) by Baole Wei and Jianguo Jiang treat words as nodes in a graph. By modeling the semantic and spatial relationships between words, the system can use the "reading order" or linguistic context to decide if two adjacent fragments belong to the same sentence or represent two different entities.

Similarly, Heqian Qiu and Hongliang Li have explored Region Prompt Tuning, using vision-language models like CLIP to provide fine-grained textual prompts that help the detector distinguish between small, densely packed characters that look visually similar but are semantically distinct.

By combining pixel-level precision with geometric and relational awareness, these researchers are transforming scene text detection from a simple "find the box" task into a sophisticated exercise in visual disentanglement.

Papers in this version

Showing 60 of 129 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 113 papers
  2. Version 1 · 2026-10-04 · 129 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_text_instance_separation_62ce8d41,
  title = {text instance separation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/text-instance-separation/rdr_62ce8d4196184f1cb831741b0781642e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_disentangling_dense_and_overlapping_scen_62ce8d41_v1,
  title = {Disentangling Dense and Overlapping Scene Text Instances},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/disentangling-dense-and-overlapping-scene-text-instances/rdr_62ce8d4196184f1cb831741b0781642e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}