Introduction
Traditional scene text detection has focused primarily on identifying individual words or lines. However, in real-world scenarios, information is often presented in complex, multi-line structures where the logical grouping of text is essential for understanding. This task, known as Contextual Text Block Detection (CTBD), aims to group fragmented text units into coherent messages.
While document layout analysis is well-established for structured PDFs, CTBD in natural scenes remains a significant challenge. Natural scene text exhibits extreme variety in fonts, orientations, and scales, often set against cluttered backgrounds. Previous approaches typically treated this as a sequence modeling problem, which often fails when layouts are non-linear or densely packed. In this paper, we present DRFormer (Dynamic Relation Transformer), a framework that reframes CTBD as a graph generation task to better capture complex spatial and logical relationships.
Methodology
DRFormer adopts a two-stage framework that treats text units as nodes and their reading order as edges within a graph. This approach allows for a more flexible representation of non-linear layouts compared to traditional sequence-based methods.
Node Detection
In the first stage, the framework utilizes a modified DQ-DETR (a state-of-the-art object detector) to identify "integral text units." These units, which may be individual words or characters depending on the dataset, serve as the foundational nodes for the subsequent graph generation.
Edge Generation: The Dynamic Relation Transformer
The core of the framework is the Dynamic Relation Transformer, which predicts the relationships (edges) between detected nodes through an iterative refinement process. Key innovations include:
- Dual Interactive Decoder: The model features a node decoder and an edge decoder that interact throughout the process. This allows the model to refine the features of the text units based on their predicted relationships and vice versa.
- Relation-Aware Self-Attention: To reduce noise from unrelated text blocks (e.g., a "STOP" sign near a "SALE" sign), the node decoder employs a relation-aware mechanism. This restricts nodes to only "attend" to other nodes within the same predicted block, preserving feature purity.
- Dynamic Edge Query Refinement: Rather than a single-pass prediction, the model iteratively prunes unlikely edges and proposes new connections for isolated nodes. This self-correcting mechanism ensures that the final graph structure is robust and logically consistent.
- Cross-Attention Priority: The authors found that performing cross-attention (incorporating image features) before self-attention (node-to-node interaction) significantly improved the model's ability to ground its predictions in the visual context.
Experimental Setup
The DRFormer was evaluated on two primary benchmarks designed for logical text grouping:
- SCUT-CTW-Context: An augmentation of the CTW-1500 dataset, focusing on word-level units in diverse natural scenes.
- ReCTS-Context: Based on the ICDAR2019-ReCTS dataset, this benchmark focuses on Chinese characters on signboards, which often feature highly complex and non-standard layouts.
Performance was measured using Local Accuracy (LA), Local Continuity (LC), and Global Accuracy (GA).
Results and Analysis
DRFormer achieved state-of-the-art (SOTA) performance across all metrics, demonstrating a significant leap over previous methodologies.
- SCUT-CTW-Context: At an IoU threshold of 0.5, DRFormer reached 47.8% Global Accuracy, outperforming the previous best method (CUTE) by over 15 percentage points.
- ReCTS-Context: The model achieved 71.8% Global Accuracy, representing a substantial improvement over baseline methods in handling complex signboard layouts.
- Upper Bound Analysis: To test the robustness of the relation prediction module independently of the detector, the model was evaluated using ground-truth text units. In this scenario, the grouping logic reached 83.9% LA on SCUT-CTW, confirming that the Dynamic Relation Transformer is highly effective at capturing logical structures when provided with accurate nodes.
Discussion and Conclusion
The success of DRFormer highlights the advantages of treating contextual text detection as a graph generation problem. By moving away from linear sequence modeling, the framework can naturally handle multi-directional reading orders and complex spatial arrangements. The iterative refinement and relation-aware attention mechanisms further ensure that the model remains robust against environmental noise.
Future work will explore the integration of "text spotting"—simultaneously detecting and recognizing text content—to determine if semantic information can further improve relationship prediction. Additionally, the authors aim to apply the principles of Dynamic Graph Structure Refinement to other graph-based tasks in computer vision and beyond.
Summary updated