TL;DR
This paper introduces "sHarris," an improved corner detection algorithm for accurately identifying text regions in video images, even under challenging conditions like low contrast and complex backgrounds. By combining an enhanced Harris corner detector with morphological operations and validation rules, the method effectively generates and refines candidate text regions, outperforming standard Harris detection.
What Problem Does This Paper Address?
Detecting and recognizing text in video images, particularly captions, is crucial for understanding video content. However, this task is complicated by several factors: captions can have varying contrasts, sizes, colors, and languages, and they often appear against complex, dynamic backgrounds. Existing methods, while addressing some of these challenges, can struggle with low-contrast text or complex backgrounds, leading to missed detections or false positives. This paper aims to improve the robustness and accuracy of caption region detection in videos.
Methods
The proposed method, termed "sHarris," breaks down caption region detection into three main steps:
- Improved Text Corner Detection (sHarris): The core innovation is an enhanced Harris corner detector. The authors introduce the concept of using the second difference of the image signal in addition to the first difference. This is motivated by the observation that second derivatives produce zero-crossings at points of high contrast, which is beneficial for edge detection, while first derivatives produce high absolute values. The paper details how this is integrated into the Harris corner detection framework, leading to the "sHarris" detector.
- Candidate Text Region Generation: After detecting corners using sHarris, morphological operations, specifically dilation, are applied. The corners are used as seeds, and repeated dilations (around twelve iterations) are performed to grow connected components. These components serve as initial candidate text regions.
- Non-text Region Elimination: To refine the candidate regions, a validation process is employed. This involves calculating the horizontal and vertical diameters of each candidate region and their ratios. Additionally, a scan line analysis is performed within the candidate regions to check for structural similarity, helping to distinguish genuine text regions from non-textual background elements.
Datasets
The paper states that "hundreds of images are randomly selected from aforementioned video clips," including television, movies, and news from various sources (TVB, CCTV, BBC, etc.) with diverse languages (Chinese, English, etc.). No specific named datasets are mentioned, nor is a new benchmark dataset introduced.
Key Insights and Ideas
- Second Difference for Corner Detection: The introduction of the second difference into the Harris corner detection algorithm is a key insight. This modification appears to enhance the detector's ability to find corners in challenging scenarios, particularly where contrast is low.
- Morphological Operations for Region Growth: Using repeated dilation from detected corners is an effective way to generate connected components that are likely to encompass text. The specific number of dilations (around twelve) suggests a carefully tuned parameter.
- Multi-faceted Validation: Relying on a combination of size ratios (horizontal/vertical diameters) and scan line similarity provides a robust mechanism for filtering out false positives. This multi-pronged approach is more likely to succeed than a single criterion.
- Robustness to Variations: The paper emphasizes that sHarris is designed to be robust to variations in text contrast, size, color, and language, which are common issues in real-world video captions.
Key Results
The paper presents a comparative analysis in Table 1, evaluating their proposed workflow using sHarris against the standard Harris detector, and two other methods ([16] and [7]).
- sHarris vs. Harris: The proposed workflow using sHarris achieved a Correctly Detected Region (CDR) rate of 92.86% and a Precision Rate (PR) of 96.16%. The standard Harris detector, in contrast, achieved 84.4% CDR and 81.8% PR. This indicates a significant improvement in both detection accuracy and the precision of the detected regions.
- Comparison with Other Methods: The sHarris workflow also outperformed Method [16] (90.48% CDR, 94.74% PR) and Method [7] (89.15% CDR, 92.84% PR), suggesting its effectiveness.
- Qualitative Results: Figures 9-14 visually demonstrate the performance. Figures 10 and 13 show that sHarris (with CT=10 and 20) produces cleaner text candidate regions and final detections, especially in complex backgrounds, compared to the Harris detector (Figure 11 and 14), which often includes background noise.
Limitations and Future Work
- Polychrome Caption Texts: The paper explicitly states that "Polychrome caption texts do not be considered in this paper" (Page 8). This is a clear limitation, suggesting that future work could extend the method to handle multi-colored captions.
- Parameter Sensitivity (CT): While the paper mentions a global threshold CT, the experimental results (Figures 10, 11, 13, 14) show that the detection performance is influenced by the value of CT. The authors suggest that CT values between 10 and 20 are optimal for their method. Further investigation into adaptive thresholding or automatic parameter tuning could be a future direction.
- Missed and False Detections: Despite the improvements, Figures 16 and 17 show examples of missed and false detections, indicating that the method is not perfect and there is room for further refinement.
- Book Cover/Other Images: The paper notes that texts in book covers or other images can be detected (Figure 18), hinting at potential broader applications beyond video captions.
Summary updated