Introduction
In professional domains such as engineering, emergency response, and military Command and Control (C4I), users must interact with vast libraries of standardized symbols. Traditional input methods rely on deep, hierarchical menus that are often slow and unintuitive, particularly in high-pressure environments. While hand-drawn sketching offers a more natural alternative, it faces significant technical hurdles: the fine-grained visual differences between professional symbol categories and the high degree of variability in individual sketching styles.
To address these challenges, this research introduces SketchIME, a comprehensive system designed to transform rough sketches into standardized professional symbols. The work contributes both a massive new specialized dataset and a novel dual-stream neural network architecture that achieves high accuracy through simultaneous recognition and semantic segmentation.
The SketchIME Dataset
Existing sketch datasets, such as Google’s Quick, Draw! or TU-Berlin, focus primarily on everyday objects. These are insufficient for professional applications where symbols are defined by precise, systematic rules.
The authors present the SketchIME Dataset, the first large-scale collection specifically for C4I symbols. It encompasses:
- 374 categories and 139 unique semantic components.
- 56,000 samples collected from 67 different participants.
Unlike general datasets, SketchIME features high intra-class similarity, where the distinction between two symbols might depend on a single stroke or a specific geometric arrangement. This requires models to learn fine-grained features rather than just general silhouettes.
Methodology: The SketchRecSeg Architecture
The core of the system is SketchRecSeg, a dual-stream architecture that treats recognition (identifying the symbol) and segmentation (identifying the constituent parts) as mutually beneficial tasks.
Dual-Stream Processing
- CNN Stream (ResNet18): This stream processes the sketch as a static image, capturing global shape and visual features.
- GNN Stream (Dynamic Graph Convolution): This stream treats the sketch as a series of points (SVG format), capturing the temporal and structural relationships between strokes.
Multilevel Supervision
A key innovation is the use of a "translation matrix" that maps categories to their semantic parts. By using recognition probabilities to supervise the segmentation stream, the model ensures that the identified parts are logically consistent with the predicted symbol category. This approach not only improves accuracy but also enhances interpretability; if a symbol is misclassified, the segmentation map reveals exactly which stroke confused the model.
Personalization and Growth
To handle user variability, the system employs Domain Adaptation (CDAN), allowing the model to adapt to a specific user's style with as few as 1–5 examples. Additionally, the system incorporates Class-Incremental Learning (FACT), enabling it to learn new symbol categories over time without forgetting existing ones or requiring a full retraining of the model.
Experimental Results
The SketchRecSeg model demonstrates state-of-the-art performance across multiple benchmarks:
- Base Performance: On the SketchIME-SRS dataset, the model achieved 77.21% Acc@1 out of the box, significantly outperforming standalone Vision Transformers (31.66%) and ResNet18 (59.10%).
- The Power of Adaptation: When utilizing 5-shot Domain Adaptation (DA5), accuracy increased to 96.21%, proving the system's effectiveness in personalized real-world settings.
- Segmentation Precision: The model achieved a 97.01% P-Metric (point-based accuracy), indicating near-perfect identification of individual symbol components.
- Cross-Dataset Versatility: The model also outperformed existing methods (like SketchGNN) on the SPG dataset, a common benchmark for sketch grouping.
Conclusion and Future Work
SketchIME represents a significant step forward in making sketch-based input viable for professional systems. By combining a specialized dataset with a dual-stream architecture that leverages both visual and structural data, the system achieves the precision required for complex symbology.
Future research will focus on transitioning from "offline" recognition to "online" recognition, providing real-time symbol suggestions as the user draws. Additionally, the authors aim to scale the system to handle the thousands of symbols found in the most complex professional libraries, further testing the limits of their class-incremental learning framework.
Summary updated