GLADDER: Combining Gesture and Geometric Sketch Recognition

Paul Corey, Tracy HammondAAAI 2008

Abstract

Sketch recognition systems typically rely on either gesture-based or geometric-based methodologies, each with inherent limitations. GLADDER (Gesture and LADDER) is a hybrid framework that integrates these two paradigms. By utilizing a rejection method based on Mahalanobis distance, the system dynamically routes input to the most appropriate recognition engine. Our evaluation shows that this integrated approach achieves 79.9% accuracy, a significant improvement over using either method in isolation.

Introduction

The field of sketch recognition—enabling computers to interpret hand-drawn shapes—has historically been divided into two primary approaches. Gesture-based recognition focuses on the temporal and stylistic aspects of a stroke (e.g., stroke order and speed), making it suitable for complex symbols like alphanumeric characters. However, it is often brittle, failing when users deviate from expected drawing styles. Conversely, geometric recognition analyzes the final visual form, decomposing shapes into primitives like lines and arcs. While robust for standard geometric shapes (e.g., rectangles), it struggles with complex, non-geometric symbols.

In practical applications, such as engineering diagrams, users frequently mix both types of input. GLADDER addresses this "either-or" limitation by combining the Modified Rubine gesture recognizer with the LADDER geometric framework into a unified system.

Methodology

The GLADDER system integrates two modified existing frameworks:

  1. Modified Rubine Recognizer: The original Rubine algorithm utilizes a linear classifier with a pooled covariance matrix. The authors enhanced this into a quadratic classifier that maintains class-specific covariance matrices. This allows for more precise classification using the Mahalanobis distance, which measures the distance of an input point from the mean of a distribution in terms of standard deviations.
  2. LADDER Recognizer: This geometric engine breaks strokes into primitives and applies a hierarchical classifier to identify shapes based on geometric constraints.

Integration Strategy

To determine which engine should process a given stroke, GLADDER employs a rejection method. Upon receiving a stroke, the Rubine classifier calculates the Mahalanobis distance to the nearest gesture class. If this distance is below an empirical threshold of 35, the system prioritizes the gesture interpretation. If the distance exceeds this threshold, the system defaults to the LADDER geometric interpretation.

Experimental Evaluation

The system was evaluated through a user study involving 23 participants. The dataset comprised 3,520 examples of 28 distinct shapes, including both geometric primitives and single-stroke mathematical symbols.

The dataset includes a mix of geometric shapes (left) and gesture-based mathematical symbols (right).

Of the collected data, 1,824 examples were used for training, with the remainder reserved for testing the system's accuracy across different input types.

Key Results

The integration of gesture and geometric engines yielded a clear performance advantage. The system effectively assigned strokes to the correct classifier type, allowing for a more natural drawing experience.

Comparison of recognition accuracy across the standalone engines and the integrated GLADDER system.

As shown in the results, the integrated GLADDER system achieved an accuracy of 79.9%, outperforming the Modified Rubine (61.9%) and LADDER (75.2%) engines when used independently. This demonstrates that the hybrid approach successfully mitigates the weaknesses of individual specialized engines.

Discussion and Future Work

A key insight from this research is the utility of the Mahalanobis distance as a confidence metric, allowing the system to "reject" unlikely gesture matches in favor of geometric analysis. This "sum greater than its parts" effect occurs because the two engines tend to fail on different types of inputs.

However, challenges remain. There is significant overlap between certain classes (e.g., a gesture for "0" versus a geometric circle), which can lead to ambiguity without higher-level context. Furthermore, the current implementation is limited to single-stroke symbols. Future research will focus on multi-stroke recognition and developing a more sophisticated tiered thresholding system to better interleave results from both engines.

Summary updated

BibTeX
@inproceedings{corey2008gladder,
  author = {Paul Corey and Tracy Hammond},
  title = {{GLADDER: Combining Gesture and Geometric Sketch Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2008},
}
Figure