Arbitrary-Shaped Scene Text Detection and Spotting

75 concepts70 papers1 proposal

Traditional object detection focuses on axis-aligned bounding boxes, but text in the wild rarely conforms to these constraints. Scene text is often oriented, curved, or distorted by the perspective of the camera. Modern research has shifted toward flexible geometric representations and the integration of linguistic context to bridge the gap between raw pixels and readable characters.

Modeling Flexible Geometry

The evolution of scene text detection is defined by how we represent the shape of a text instance. Early methods adapted standard object detectors like Faster R-CNN, but these were limited by horizontal boxes. Liao et al. addressed this in TextBoxes++ by using quadrilateral representations and specialized "text-box" layers with varied aspect ratios, allowing the model to capture slanted and elongated text more naturally.

To handle more extreme deformations, such as curved or "inverse" text, researchers have moved toward parametric curves. Qiao et al. introduced DNTextSpotter, which uses Bezier control points to define text boundaries. By applying denoising training to these positional queries, the model achieves high stability even when text is oriented in unusual directions. Similarly, Liu et al. proposed PBFormer, which utilizes a "Polynomial Band" representation. This approach captures complex shapes using a fixed number of parameters within a transformer architecture, enabling the system to detect crowded and curved text without the need for extensive pre-training.

Bottom-Up Assembly and Instance Segmentation

Beyond global shape regression, another successful strategy involves treating text detection as a grouping problem. Instead of predicting a whole box at once, these methods identify small components and assemble them.

  • Component Assembly: Li et al. developed AT-Text, which uses a multi-scale feature segmentation model to generate text masks and then employs a bottom-up assembly process to handle dense, multi-language text.
  • Character Embedding: Li et al. proposed CENet, which simultaneously learns character locations and their embeddings. This turns the detection of irregular text into a clustering task, where characters belonging to the same word are grouped by their proximity in the embedding space.
  • Topological Maps: Zhu et al. introduced TextMountain, which frames detection as instance segmentation. By predicting a "center-border probability" map—where the center of a text line acts as a "peak"—the model can effectively separate distinct text instances even when they are tightly packed.

Fusing Vision with Language

Detection is often just the first step; "text spotting" refers to the end-to-end process of detection and recognition. A key insight in recent work is that visual features alone are often insufficient for ambiguous or fragmented text.

The authors of TiCLS demonstrate that tightly coupling a character-level pretrained language model (PLM) with visual features significantly improves accuracy. By fusing linguistic insights with visual cues, the system can "read" text that is partially obscured or rotated by relying on the statistical likelihood of character sequences. This linguistic grounding is particularly effective in lexicon-free settings where the model cannot rely on a predefined dictionary.

Robustness in Specialized Domains

The principles of scene text detection are also being adapted for high-stakes or high-volume applications. For industrial-scale needs, Borisyuk et al. developed Rosetta, a system that processes hundreds of millions of images daily by combining a Faster R-CNN detector with a fully convolutional recognition model trained via curriculum learning.

In the realm of historical documents, where reading order is often non-linear, Ma et al. introduced a Reading Order Detection (ROD) module. This module mimics human visual perception to infer the correct sequence of text elements from raw image features, ensuring that once text is detected, it is interpreted in its intended context. Similarly, for challenging medieval manuscripts, Liwicki et al. showed that combining deep learning segmentation with traditional seam-carving algorithms can achieve near-perfect text-line extraction by treating segmentation as a denoising step.

Key Foundational Work

Text island spotting in large speech databases (2007) — contributed 1 concept elements to this direction.
Advances in domain independent linear text segmentation (2000) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_state_of_the_art_scene_text_detection_4f54b800,
  title = {state-of-the-art scene text detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/state-of-the-art-scene-text-detection/rdr_4f54b8008f3a4b14bd1c3fd5be0b518a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_state_of_the_art_scene_text_detection_4f54b800_v1,
  title = {state-of-the-art scene text detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/state-of-the-art-scene-text-detection/rdr_4f54b8008f3a4b14bd1c3fd5be0b518a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake