End-to-end Neuromorphic Lip Reading

Hugo Bulzomi, Marcel Schweiker, Amélie Gruel, Jean MartinetWorkshops @ CVPR 2023 · DOI

Introduction

Lip reading is a computationally intensive task traditionally dominated by frame-based computer vision. These systems require significant power to process full image frames at fixed intervals, making them difficult to deploy on battery-constrained assistive devices like hearing aids. This paper introduces the first end-to-end neuromorphic lip-reading system, leveraging event-based cameras and Spiking Neural Networks (SNNs) to achieve high efficiency without sacrificing significant accuracy.

The Neuromorphic Pipeline

Unlike traditional cameras, event-based sensors only record pixel-level brightness changes, producing a sparse stream of "events." To process this data while maintaining the benefits of deep learning, the researchers developed a spiking adaptation of the Multi-grained Spatio-Temporal Features Perceived (MSTP) network.

The pipeline begins with event preprocessing, where asynchronous events are organized into a 3D voxel grid. This synchronous representation allows the use of surrogate gradient descent during training. Spatial features are then extracted using a Spiking Element-Wise ResNet (SEW-ResNet) backbone.

Temporal Modeling with Stateful Synapses

A critical challenge in SNNs is capturing long-term temporal dependencies. While traditional networks use Gated Recurrent Units (GRUs), the authors found that "spiking GRUs" were inefficient for this task. Instead, they utilized stateful synapses. These components accumulate input spikes over time, maintaining a "membrane potential" that effectively bridges the gap between rapid spatial features and slower word-level classification.

Temporal integration in a spiking neuron: discrete input spikes (top) are accumulated into a continuous membrane potential (bottom).

By using Parametric Leaky-Integrate-and-Fire (PLIF) neurons with learnable time constants, the network can adapt its temporal memory during the training process to better suit the dynamics of human speech. Replacing a simple fully connected layer with these stateful synapses improved accuracy from 52.2% to 60.2%.

Optimization via Surrogate Gradients

Training SNNs is difficult because the spiking threshold function is non-differentiable. To overcome this, the authors employed surrogate gradient descent. The choice of surrogate function—which approximates the gradient during the backward pass—is vital for model convergence.

Comparison of surrogate gradient functions (Erf, Piecewise quadratic, and ATan) used to approximate the non-differentiable spike activation during training.

Experimental results indicated that the Gaussian error surrogate function (Erf) provided superior performance for lip-reading tasks compared to other common alternatives like ATan or Piecewise Quadratic functions.

Performance and Efficiency

The Spiking MSTP model achieved a top accuracy of 60.2% on the DVS-Lip dataset. While this is lower than the 72.1% achieved by the original frame-based ANN, the neuromorphic approach offers a massive leap in efficiency. The SNN model requires only 47MB of memory—a 5x reduction compared to the 241.5MB required by the ANN. This efficiency-accuracy trade-off makes it a prime candidate for edge computing and portable assistive technology.

Conclusion

This work demonstrates that a fully neuromorphic pipeline can perform complex spatio-temporal tasks like lip reading. By replacing recurrent units with stateful synapses and optimizing with appropriate surrogate gradients, the researchers closed the gap between bio-inspired efficiency and state-of-the-art performance. Future work will focus on closing the remaining 12% accuracy gap and scaling the system to handle full sentence-level recognition in real-world environments.

Summary updated

BibTeX
@inproceedings{bulzomi2023end,
  author = {Hugo Bulzomi and Marcel Schweiker and Amélie Gruel and Jean Martinet},
  title = {{End-to-end Neuromorphic Lip Reading}},
  booktitle = {Workshops @ CVPR},
  year = {2023},
  doi = {10.1109/cvprw59228.2023.00431},
}
Figure