MobiVSR : Efficient and Light-Weight Neural Network for Visual Speech Recognition on Mobile Devices

Nilay Shrivastava, Astitwa Saxena, Yaman Kumar, Rajiv Ratn Shah, Amanda Stent, Debanjan Mahata, Preeti Kaur, Roger ZimmermannInterspeech 2019 · OpenReview

Abstract

Visual speech recognition (VSR) is the task of recognizing spoken language from video input only, without any audio. VSR has many applications as an assistive technology, especially if it could be deployed in mobile devices and embedded systems. The need for intensive computational resources and large memory footprint are two major obstacles in deploying neural network models for VSR in a resource constrained environment. We propose a novel end-to-end deep neural network architecture for word level VSR called MobiVSR with a design parameter that aids in balancing the model’s accuracy and parameter count. We use depthwise 3D convolution along with channel shuffling for the first time in the domain of VSR and show how it makes our model efficient. MobiVSR achieves an accuracy of 70% on a challenging Lip Reading in the Wild dataset with 6 times fewer parameters and 20 times smaller memory footprint than the current state of the art. MobiVSR can also be compressed to 6 MB by applying post training quantization.

Introduction: The Challenge of Mobile VSR

Visual Speech Recognition (VSR), often referred to as automated lip-reading, is a critical assistive technology for individuals unable to produce sound or for communication in high-noise environments. However, state-of-the-art (SOTA) VSR models are traditionally heavy, relying on massive 3D convolutional neural networks (CNNs) and Recurrent Neural Networks (RNNs) that often exceed 130MB.

Deploying such models on mobile devices presents significant hurdles:

  • Memory Constraints: High RAM consumption can lead to OS-level process termination.
  • Storage Footprint: Large models increase application size, hindering distribution.
  • Energy Consumption: Intensive computation and frequent DRAM access rapidly deplete battery life.

MobiVSR addresses these challenges by introducing a "mobile-first" architecture designed to balance accuracy with extreme efficiency.

The MobiVSR Architecture

The authors reimagined the VSR pipeline by replacing computationally expensive components with lightweight alternatives.

The end-to-end MobiVSR architecture, featuring a front-end with 3D convolutions, a stack of residual subgraphs, and a 1D convolutional backend.

Key Innovations:

  • Depthwise 3D Convolutions & Channel Shuffling: Profiling indicates that 3D convolutions are the primary bottleneck in video processing. MobiVSR adapts the ShuffleNet approach to 3D space, decomposing heavy convolutions into depthwise and pointwise components. Channel shuffling ensures cross-group information flow at a fraction of the original computational cost.
  • LipRes Blocks: The core of the network utilizes residual subgraphs. A hyperparameter, $\alpha$, allows for tuning the model depth, enabling developers to scale the architecture based on the target hardware's capabilities.
  • Temporal Convolutions: Unlike sequential RNNs (e.g., LSTMs), 1D temporal convolutions are highly parallelizable, leading to lower latency on mobile hardware.
  • Post-Training Quantization: By applying 8-bit non-uniform quantization, the model size is further reduced to approximately 6MB.

Experimental Evaluation

MobiVSR was evaluated on the Lip Reading in the Wild (LRW) dataset, which contains 500 words with 1,000 utterances each. The dataset is particularly challenging due to high variance in lighting, head poses, and speaker diversity, serving as a robust proxy for real-world mobile environments. Preprocessing involved extracting a $96 \times 96$ pixel Region of Interest (ROI) centered on the mouth.

Results and Efficiency

MobiVSR demonstrates that competitive performance is achievable without massive resource overhead.

Performance comparison of MobiVSR variants against the SOTA and LRW baseline.

As shown in the table above, MobiVSR-1 achieves 70.1% Top-1 accuracy, significantly outperforming the LRW baseline (61.0%). While trailing the SOTA's 83% accuracy, MobiVSR offers a dramatic improvement in efficiency:

  • 6x fewer parameters (4.4M vs. 25.1M).
  • 20x smaller storage footprint (14.2MB vs. 130MB).
  • 27x reduction in computation (10.75B vs. 290B FLOPs).

Conclusion and Future Directions

MobiVSR proves that VSR can be made viable for resource-constrained devices. The primary remaining challenge is viseme confusion, where visually similar words (e.g., "bring" vs. "being") are frequently misidentified. Future work will explore multi-modal integration—combining lightweight visual features with audio when available—and architectural refinements to capture the subtle internal mouth movements necessary to resolve these ambiguities.

Summary updated

Abstract

Visual speech recognition (VSR) is the task of recognizing spoken language from video input only, without any audio. VSR has many applications as an assistive technology, especially if it could be deployed in mobile devices and embedded systems. The need for intensive computational resources and large memory footprint are two major obstacles in deploying neural network models for VSR in a resource constrained environment. We propose a novel end-to-end deep neural network architecture for word level VSR called MobiVSR with a design parameter that aids in balancing the model’s accuracy and parameter count. We use depthwise 3D convolution along with channel shuffling for the first time in the domain of VSR and show how it makes our model efficient. MobiVSR achieves an accuracy of 70% on a challenging Lip Reading in the Wild dataset with 6 times fewer parameters and 20 times smaller memory footprint than the current state of the art. MobiVSR can also be compressed to 6 MB by applying post training quantization.
BibTeX
@inproceedings{shrivastava2019mobivsr,
  author = {Nilay Shrivastava and Astitwa Saxena and Yaman Kumar and Rajiv Ratn Shah and Amanda Stent and Debanjan Mahata and Preeti Kaur and Roger Zimmermann},
  title = {{MobiVSR : Efficient and Light-Weight Neural Network for Visual Speech Recognition on Mobile Devices}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=TW7Sr59a66F},
}
Figure