Continuous Interaction with A Smart Speaker via Low-Dimensional Embeddings of Dynamic Hand Pose

Songpei Xu, Chaitanya Kaul, Xuri Ge, Roderick Murray-Smith

Introduction

Traditional mid-air gesture systems often struggle with a trade-off between complexity and responsiveness. Many deep learning models require long video sequences (e.g., 30 frames) to recognize a gesture, leading to noticeable lag that frustrates users. Furthermore, most systems only support "discrete" gestures—simple triggers like "play" or "pause"—rather than "continuous" interaction, such as sliding through a color gradient or exploring a musical landscape.

Researchers have developed a low-latency system for controlling smart speakers using continuous mid-air hand gestures. By jointly training an autoencoder and a PointNet classifier, the system maps hand poses into a 2D "pose space" that allows users to intuitively navigate a massive library of 55,000 music tracks based on mood and emotion with just 2.4ms of processing delay.

Methodology

The proposed pipeline, illustrated in Figure 1, consists of several interconnected stages designed for real-time performance.

The proposed pipeline for mid-air gesture control, showing the flow from raw frames to the joint learning architecture and final music space mapping.
  1. Feature Extraction: The system uses MediaPipe Hands to extract 3D coordinates for 21 hand landmarks from standard video frames.
  2. Dimensionality Reduction: A fully-connected Autoencoder compresses these high-dimensional landmarks into a 2D "Latent Feature" or Pose Space.
  3. Joint Learning: Unlike standard approaches that train an autoencoder and a classifier separately, this model uses Joint Learning. It simultaneously minimizes reconstruction loss (to keep the 2D map accurate) and classification loss (to ensure different gestures are easily distinguishable in that map).
  4. Classification: A 2D PointNet model classifies the gestures. Crucially, the authors found that using a sequence of just 2 frames provided the best balance between accuracy and near-instantaneous feedback.
  5. Music Space Mapping: To make the interaction meaningful, the researchers took a dataset of 55,000 music tracks (characterized by 34 features like genre and emotion) and used UMAP to project them into a 2D Music Space. The user's 2D hand pose is then scaled and mapped directly onto this music space.

The system was trained on a custom dataset of 60,225 frames across 6 gesture types (Continuous arm open/close, Circle clockwise/counterclockwise, Pinch, and Double pinch) collected from 7 volunteers.

Key Insights: The Power of Joint Learning

Standard unsupervised methods like UMAP or basic autoencoders often produce "messy" clusters where different gestures overlap. By adding a classification objective to the autoencoder's training, the model learns to "push" different gesture clusters apart, making the interaction much more robust. This is clearly visible in the comparison of embedding methods (Figure 2).

Comparison of embedding methods. Row (a) shows UMAP, row (b) shows a standard autoencoder, and rows (c/d) show the joint autoencoder-classifier, which produces much cleaner separation between gesture types.

Beyond architecture, the researchers found that 2 frames are sufficient for high-precision gesture recognition in this context, minimizing the "interaction response delay" that typically affects user experience. To prevent the system from being confused by different hand sizes or distances from the camera, the authors utilized quaternion representations to stabilize the embedding.

Key Results

  • Speed: The total inference time for the entire process (autoencoder + classifier + drawing) is approximately 2.4ms, enabling true real-time interaction.
  • Accuracy: The 2-frame PointNet model achieved 75.2% precision and 62.5% mAP, outperforming the 1-frame version while maintaining much lower latency than an 8-frame version.
  • User Exploration: Qualitative tests showed that users could successfully navigate the music space to find specific moods (e.g., moving from "tender" to "erotic" zones) simply by varying their hand pose. Continuous hand pose tracks were recorded as users explored the musical landscape.

Conclusion

While the current system shows promise for continuous interaction, future work will focus on testing performance in cluttered home environments and expanding the gesture vocabulary. Currently, the model provides a low-latency, interpretable interface that bridges the gap between physical movement and digital content exploration.

Summary updated

BibTeX
@inproceedings{xu2023continuous,
  author = {Songpei Xu and Chaitanya Kaul and Xuri Ge and Roderick Murray-Smith},
  title = {{Continuous Interaction with A Smart Speaker via Low-Dimensional Embeddings of Dynamic Hand Pose}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10096097},
}

Loading related content...

Figure