TL;DR
Hierarchical Text Spotter (HTS) unifies text spotting and geometric layout analysis into a single framework that recovers a complete 4-level text hierarchy: characters, words, lines, and paragraphs. By shifting from traditional word-centric detection to a line-and-character-centric approach, HTS achieves state-of-the-art results on both curved and highly dense text benchmarks using a single model without dataset-specific fine-tuning.
What Problem Does This Paper Address?
Most existing text spotting methods are "word-centric"—they extract text at the word level by treating words as isolated sequences of characters. This approach completely ignores the broader geometric layout, such as how words group into lines and paragraphs. Conversely, layout analysis has traditionally been treated as a separate task altogether, mostly confined to clean document images.
This disconnect creates a major challenge for downstream tasks like text-based Visual Question Answering (VQA) or scene understanding, where reading the text is only half the battle; you also need to understand its structural context. Furthermore, word-centric models struggle significantly in dense text environments (like menus or signs) where words are tightly packed and varied in scale.
Methods
HTS tackles this with a two-stage pipeline that unifies detection and recognition hierarchically:
- Unified-Detector-Polygon (UDP): Instead of searching for words, UDP detects entire text lines using Bezier Curve polygons. It simultaneously generates an affinity matrix to cluster these lines into paragraph groups. To handle highly diverse text shapes and sizes, the authors introduce a Location and Shape Decoupling Module (LSDM). LSDM splits the task into two parallel heads: one predicts an Axis-Aligned Bounding Box (AABB) for location, and the other predicts local Bezier control points normalized within that box.
- Line-to-Character-to-Word (L2C2W) Recognizer: Once lines are cropped and rectified, this Transformer-based encoder-decoder model jointly predicts character classes and character-level bounding boxes. Crucially, it is trained to predict the standard space character as a delimiter, allowing it to seamlessly merge character sequences back into word-level entities.
Datasets
The model was trained and evaluated on a variety of standard and dense text datasets:
- Real-image Datasets: HierText (highly dense text, averaging over 100 words per image), Total-Text (curved text), ICDAR 2015 (horizontal scene text), and CTW1500.
- Synthetic Datasets: SynthText and Synth90K, along with an internal dataset of 1M synthetic text lines used to provide character-level bounding box annotations during training.
Key Insights and Ideas
- Line-First, Word-Later: In dense text environments, detecting individual words directly is error-prone due to crowding. Detecting lines first provides more robust context, and leveraging the recognizer to find spaces is a much cleaner way to segment words.
- Decoupling Geometry: Standard networks that directly regress global Bezier control points using L1 loss often fail on complex datasets because the loss is dominated by coordinate location rather than shape. Normalizing the shape coordinates relative to a local bounding box (via LSDM) allows the network to learn intricate text curvatures far more effectively.
Key Results
HTS achieved state-of-the-art performance across multiple distinct benchmarks without any dataset-specific fine-tuning:
- Text Spotting: On the curved-text Total-Text benchmark, HTS achieved an F1-score of 85.17%. On the challenging, dense HierText dataset, HTS achieved a word-level F1-score of 76.52%, vastly outperforming the previous best word-centric model (GLASS at 64.47%).
- Layout Analysis: Evaluated on HierText, HTS outperformed the baseline Unified Detector, boosting Panoptic Quality (PQ) by +4.08 on line grouping and +6.65 on paragraph grouping.
Limitations and Future Work
- Latency: Running at 5.6 FPS on HierText and 7.8 FPS on ICDAR 2015 (measured on an NVIDIA A100), the model is currently too slow for real-time applications. The authors suggest that sharing features between the detector and recognizer to make the pipeline fully end-to-end could reduce computation.
- Annotation Dependency: Training the UDP detector requires line-level annotations, which are rarer in public datasets than word-level boxes.
- Character Evaluation: Because public benchmarks lack character-level bounding box ground truths, the authors could only evaluate character localization accuracy indirectly via word-spotting performance.
Summary updated