Active Gesture Recognition using Learned Visual Attention

Trevor Darrell, Alex PentlandNeurIPS 1995 · OpenReview

Introduction

Researchers at the MIT Media Lab have developed a system that uses an active, steerable camera to recognize human gestures by "learning where to look." By framing visual attention as a reinforcement learning problem—specifically a Partially Observable Markov Decision Process (POMDP)—the system learns to move its narrow-field "foveal" camera between a user's head and hands. This allows it to gather the specific high-resolution details needed to distinguish between different gestures in an unconstrained environment.

The Challenge of Active Vision

Traditional computer vision systems often struggle in unconstrained environments, like a typical office, where a user moves freely. Recognizing subtle signals, such as facial expressions or hand gestures, requires high-resolution images. However, a single wide-angle camera doesn't provide enough detail, and a fixed narrow-angle camera will quickly lose the user as they move.

The core challenge is active vision: if you have a steerable camera that can zoom in on specific body parts, how do you decide where to point it at any given moment? This is the "attention" problem. If a user is performing a gesture that involves both a specific hand shape and a facial expression, the system must learn an optimal sequence of "foveation" (looking) actions to maximize its confidence before making a final classification.

Methods and System Architecture

The authors treat gesture recognition as an Active Gesture Recognition (AGR) task, modeled as a POMDP. Because the camera can only see one thing at a time (e.g., the face OR the left hand), the full state of the user is "hidden" from the system at any single moment.

Dual-camera active vision setup

Key technical components of the system include:

  • Dual-Camera Setup: A wide-field camera tracks the user's general 3D position (using the ALIVE system), while a narrow-field camera on an active pan/tilt base zooms in on salient parts like the face or hands.
  • Instance-Based Reinforcement Learning: Instead of a standard state table, the researchers use a "Nearest Sequence Memory" (NSM) approach. The system stores a history of action-observation tuples and evaluates the current situation by looking at the $K$ most similar sequences from its past.
  • Multiple Model Q-Learning: This is the paper's primary algorithmic contribution. Standard Q-learning struggled to converge when trying to identify multiple different target gestures simultaneously. To solve this, the authors proposed using a separate reinforcement learning agent for each target gesture. Each agent maintains its own Q-values (confidence) based on shared observations, and the system selects the action from the agent with the highest current Q-value.

Experimental Setup

The research predates modern benchmark datasets, so evaluation was performed using the ALIVE System, an interactive "Magic Mirror" environment developed at MIT. The experimental setup included a set of 8 target gestures involving combinations of facial expressions (neutral, smile, surprise) and hand poses (neutral, point, grab). Non-target states were also included as distractors to train the system to be selective.

Key Insights

  • Attention as Action: The most profound idea presented is that "looking" is a functional action that can be optimized via rewards, just like a robot moving its arm.
  • Decomposing Complexity: By using Multiple Model Q-learning, the authors demonstrated that a complex multi-class classification problem can be broken down into several simpler "is this my target?" problems running in parallel.
  • History Matters: In partially observable environments, the current frame isn't enough. By using a history-based approach, the system uses the sequence of what it saw previously to disambiguate what it is seeing now.

Key Results

The system was tested on a task with 8 distinct gesture targets. The researchers measured the classification error rate relative to the number of "nearest neighbors" ($K$) used in the learning algorithm.

Classification error rate relative to K nearest neighbors
  • With $K=1$ (looking at only the single most similar past experience), the error rate was high at nearly 59.6%.
  • As $K$ increased, performance improved dramatically. At $K=2$, the error dropped to 8.68%, and at $K=4$, it fell to just 0.84%.
  • For $K > 2$, the system achieved near-perfect recognition, demonstrating that the multiple-model approach successfully learned the optimal foveation strategies to discriminate between gestures.

Limitations and Future Work

  • Scaling the History: The instance-based approach requires storing and searching through a history of observations. As the system runs longer, the computational cost of finding nearest neighbors could become a bottleneck.
  • Simple Gesture Vocabulary: While effective for the 8 tested gestures, the paper doesn't explore how the system would handle highly dynamic, fluid gestures where timing is more critical than static poses.
  • Single User Focus: The system was designed for an environment with a single active user. Coordinating attention in a crowded room with multiple people would require significantly more complex action selection.

Summary updated

Abstract

We have developed a foveated gesture recognition system that runs in an unconstrained office environment with an active camera. Us(cid:173) ing vision routines previously implemented for an interactive envi(cid:173) ronment, we determine the spatial location of salient body parts of a user and guide an active camera to obtain images of gestures or expressions. A hidden-state reinforcement learning paradigm is used to implement visual attention. The attention module selects targets to foveate based on the goal of successful recognition, and uses a new multiple-model Q-Iearning formulation. Given a set of target and distractor gestures, our system can learn where to foveate to maximally discriminate a particular gesture.
BibTeX
@inproceedings{darrell1995active,
  author = {Trevor Darrell and Alex Pentland},
  title = {{Active Gesture Recognition using Learned Visual Attention}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1995},
  url = {https://openreview.net/forum?id=HkW8CUb_WB},
}
Figure