Introduction
Mainstream Scene Text Recognition (STR) systems predominantly rely on standard RGB cameras. While these sensors perform well under ideal lighting and static conditions, they often fail in "edge cases" such as high-speed motion (causing severe blur), low-light environments (introducing noise), and scenes with high dynamic range.
To address these limitations, researchers are turning to event cameras (neuromorphic sensors). Unlike traditional cameras that capture full frames at fixed intervals, event cameras record asynchronous "events"—pixel-level changes in brightness. This mechanism provides microsecond-level temporal resolution and a massive dynamic range, making them ideal for challenging STR scenarios. However, the adoption of event-based STR has been hindered by the lack of large-scale, high-resolution benchmarks and algorithms capable of processing sparse event data.
The EventSTR Dataset
This paper introduces EventSTR, the first large-scale, high-definition benchmark for event-based STR. The dataset comprises 9,928 samples at a resolution of 1280 × 720 pixels. It includes a diverse mix of Chinese and English characters captured under various challenging conditions, including low light, motion blur, occlusions, and different text orientations. To further validate the proposed methods, the authors also generated simulated datasets, WordArt and IC15, by converting existing RGB STR datasets into event streams using the ESIM simulator.
The SimC-ESTR Framework
The authors propose SimC-ESTR (Simple and Contextual Event-based STR), a framework designed to leverage the strengths of Large Language Models (LLMs) for text recognition. The architecture follows a multi-stage pipeline:
- Event Stacking: Event streams are stacked into frames to make them compatible with standard deep learning architectures.
- Visual Encoding & Alignment: An EVA-CLIP visual encoder extracts features, which are then processed by a Q-former module to align visual tokens with the input space of the LLM.
- Memory Module (MM): This module stores learnable "memory patterns" that augment visual features, helping the model reconstruct text even when event data is noisy or sparse.
- Glyph Error Correction Module (GECM): Recognizing that LLMs may confuse visually similar characters (visual homoglyphs), the GECM uses a database of glyphs and contextual validation to refine predictions.
- LLM Backbone: A fine-tuned Vicuna-7B model generates the final text output based on the aligned visual tokens and corrective prompts.
Key Results and Insights
The SimC-ESTR framework demonstrates significant improvements over traditional STR baselines. On the EventSTR dataset, it achieved a BLEU-1 score of 0.638, substantially outperforming established models like PARSeq (0.450) and MGP-STR (0.427).
Ablation studies highlighted the importance of the Glyph Error Correction Module, which alone contributed a 4.5% improvement in BLEU-1 scores. This suggests that targeted linguistic and visual correction is vital for interpreting the unique data produced by neuromorphic sensors. Furthermore, the model showed strong generalization on simulated datasets, achieving 65.1% accuracy on WordArt*.
Conclusion and Future Directions
Event cameras offer a robust solution to the "blur" and "low-light" problems in STR by capturing sharp edges of moving text. While SimC-ESTR sets a new state-of-the-art, the authors note that the reliance on a 7-billion parameter LLM poses challenges for real-time deployment on edge devices. Future work will focus on knowledge distillation to create lightweight "student" models and exploring more efficient, low-latency event representations.
Summary updated