Classifying Patterns of Visual Motion - a Neuromorphic Approach

Jakob Heinzle, Alan StockerNeurIPS 2002

Introduction

Recognizing continuous temporal patterns—such as a specific hand gesture or a moving object's path—is computationally expensive for traditional computer architectures. Standard neural networks, like Hopfield networks, often struggle with these tasks unless the patterns are periodic or have very clearly defined start and end points. Conversely, purely feed-forward architectures tend to grow unmanageably large when trying to account for the dimension of time.

To address this, researchers have developed a brain-inspired (neuromorphic) system that classifies complex visual motion patterns in real-time using analog hardware principles. By converting continuous motion into discrete "events" and using time-delays to synchronize them, the system can recognize hand-drawn gestures with high efficiency and robustness to speed variations. The goal is to create a system simple enough for low-power analog VLSI (aVLSI) hardware, yet robust enough for real-world applications like contactless man-machine interfaces for mobile devices.

System Architecture

The system's architecture is divided into three distinct neuromorphic stages that process visual input sequentially:

Overall System Architecture
  1. The Optical Flow Chip: An aVLSI front-end computes 2D global visual motion in real-time. It outputs two analog voltages representing the $x$ and $y$ components of the motion vector.
  2. Direction Selective Network (DSN): This stage uses a Winner-Take-All (WTA) architecture to map continuous motion into discrete "motion events." It divides the motion space into distinct regions (North, East, South, West, and Zero). Because only one unit "wins" at a time, the continuous input is effectively discretized.
  3. Sequence Classification Network (SCN): Serving as the "brain" of the classifier, this network uses specific time-delays to concentrate information in time. It transforms a temporal sequence into a simultaneous spatial input that a subsequent WTA network can easily identify.

Additionally, the authors introduced a Learning Rule using competitive synapse units. This allows the system to learn new sequences by adjusting weights through a slow-acting competitive process, eliminating the need for manual programming for every new gesture.

Key Mechanisms and Insights

Quasi-Discretization

One of the most powerful ideas in this approach is mapping continuous, noisy motion into discrete "events." This drastically reduces the complexity of the downstream classification task without losing the essential "meaning" or trajectory of the gesture.

Temporal Concentration

Instead of relying on complex recurrent memory architectures, the system uses simple physical time-delays.

Sequence Classification and Temporal Concentration

By delaying the signals from early parts of a sequence (e.g., the "North" part of a gesture), the network ensures they arrive at the classification unit at the exact same time as the later parts (e.g., the "West" and "East" parts). This transforms the problem of remembering a sequence into a much simpler problem of detecting a coincidence in time.

Universal WTA Architecture

Almost the entire system—from direction selection to sequence recognition and even the learning rule—is built on the Winner-Take-All principle. This consistency makes the design highly suitable for a single-chip silicon implementation.

Evaluation and Results

Because the focus is on real-time hardware interaction, the paper evaluates the system using simulated trajectories (Sum-of-Gaussians motion paths to test robustness against speed changes) and real-world hand gestures performed in front of the optical flow chip.

  • Robustness to Time Warping: The system proved remarkably resilient to speed changes. For a three-event sequence, a gesture can be stretched in time by up to 40% and still be correctly classified.
  • Alphabet Recognition: The researchers successfully demonstrated a contactless interface where a user could "write" letters in the air. The system used a defined set of 36 valid sequences (three motion events each) to encode the entire alphabet. For example, the letter 'H' was reliably mapped to a specific sequence of North, East, and South movements.
  • Hardware Efficiency: While the simulated version ran on a modest 166MHz Pentium PC, the low complexity of the network (requiring only 31 units for the man-machine interface) makes a complete "system-on-chip" aVLSI implementation highly realistic and power-efficient.

Limitations and Future Directions

Despite its efficiency, the current architecture has a few constraints:

  • Sequence Ambiguity: The model cannot easily distinguish between sequences with repeated adjacent events (e.g., N-W-E-W vs. N-W-W-W) because the integrated delay functions overlap. Forbidding adjacent identical events is a necessary constraint for the current setup.
  • Resolution: The system currently relies on only four cardinal directions. Adding more directions or differentiating speed levels (e.g., "Fast North" vs. "Slow North") would allow for richer trajectory classification.
  • Learning Constraints: Sequences must differ by more than one event to be reliably learned by different units; otherwise, one unit might "overwrite" another during the competitive learning process.

Summary updated

BibTeX
@inproceedings{heinzle2002classifying,
  author = {Jakob Heinzle and Alan Stocker},
  title = {{Classifying Patterns of Visual Motion - a Neuromorphic Approach}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2002},
}
Figure