Event-Based Dynamic Lip Region Estimation During Speech

Hamze Hammami, Omar Eldardeer, Francesco ReaarXiv 2026 · arXiv

TL;DR

This paper introduces a novel method for estimating the lip region directly from raw event camera data using a two-pathway continuous-time recurrent network (CT-RNN). By leveraging learned time constants to process event streams, the model achieves significantly higher temporal stability and better generalization to real-world sensors compared to standard convolutional baselines, even when trained only on simulated data.

What Problem Does This Paper Address?

Event cameras are bio-inspired sensors that report per-pixel brightness changes asynchronously, offering microsecond temporal resolution and high dynamic range. These traits are ideal for tracking rapid facial movements like speech, especially in low-light conditions where traditional RGB cameras fail.

However, most existing event-based speech applications (like lip reading or voice activity detection) assume the mouth region has already been cropped. Current localization methods often rely on frame-based detectors or return only sparse landmarks, creating a "mismatch between application and localization." Furthermore, these methods often lack temporal stability—the predicted region might jump erratically between frames, which is detrimental for downstream analysis. This paper aims to provide a robust, "event-native" front-end that predicts a dense lip heatmap from a full, uncropped event stream.

Methods

The authors propose a specialized architecture designed to handle the unique temporal nature of event data:

  1. Polarity-Separated Voxel Grid: Events are grouped into 16 temporal bins on a $32 \times 32$ spatial grid. Crucially, "ON" and "OFF" events are kept in separate channels to preserve the signed edge structure of moving contours.
  2. Two-Pathway CT-RNN: Instead of a standard RNN, the model uses a Continuous-Time Recurrent Network (CT-RNN). This allows the network to account for the real elapsed time ($\Delta t$) of each bin using learned, log-space bounded time constants ($\tau$).
  3. Fast Pathway: Uses short time constants (1ms to 150ms) and is reset every window to follow immediate motion.
  4. Slow Pathway: Uses longer time constants (20ms to 2s) and maintains state across windows to "anchor" the prediction and provide stability.
  5. Multi-Head Output: The model predicts a lip heatmap, an uncertainty map (to identify where the "teacher" models disagree), and an auxiliary map for deep supervision.
Figure 1. Overview of the proposed event-based lip-localization architecture.

The structure of the individual CT-RNN cells, shown below, highlights how the candidate activation $f$ is blended with the previous state $h$ based on the elapsed time and the learned decay constant.

Figure 2. Structure of the continuous-time cell.

Datasets

  • Simulated Data: The primary training set consists of 12 speakers recorded at 30 fps. These were upsampled to 320 Hz using frame interpolation (VID2E) and converted into events using the ESIM simulator.
  • Real Event Data: For evaluation, the authors used Faces in Event Streams (FES) and FACEMORPHIC. These provide real-world event data, though they are not specifically designed for lip reading and often lack perfectly aligned ground truth.

Key Insights and Ideas

  • Stability vs. Precision Trade-off: Standard CNNs are often more precise on a per-frame basis but suffer from high "jitter" and frequent target switches. The proposed CT-RNN prioritizes temporal consistency, which is often more valuable for continuous speech analysis.
  • The Power of Two Timescales: The dual-pathway approach mimics biological systems. The fast pathway provides responsiveness to new movements, while the slow pathway prevents the model from "getting lost" during brief pauses or noisy intervals.
  • Generalization through Simulation: Despite being trained only on simulated events, the CT-RNN generalized significantly better to real event sensors than the CNN baseline. This suggests that learning temporal dynamics is more robust to sensor noise and contrast threshold shifts than learning static spatial appearances.

Key Results

The proposed model (with only ~29k parameters) was compared against much larger CNN and U-Net baselines (~165k parameters):

  • Temporal Stability: The CT-RNN switched targets only 2.1 times per second, compared to 8.9 for the CNN and 11.0 for the U-Net.
  • On-Lips Rate: On unseen simulated speakers, the model reached a 69.5% on-lips rate. While the CNN was higher (75.6%), it was far less stable.
  • Real-World Performance: On the FES real-event dataset, the proposed model outperformed the CNN significantly, reaching a 58.0% on-lips rate versus the CNN's 43.2%.
Figure 3. On-lips rate against switches per second.

Limitations and Future Work

  • The "Memory Trap": A key weakness of the stateful slow pathway is that if the memory commits to a wrong region (e.g., an eye), it tends to stay there. The authors suggest a future recovery mechanism is needed.
  • Scale Sensitivity: The model struggled when the face scale changed significantly (e.g., a speaker standing much closer to the camera than in the training set).
  • Simulation Gap: While generalization was good, the "on-lips" rate still drops when moving from simulated to real sensors, indicating a need for more diverse real-world training data.

Future work may involve combining this localized ROI with the pathway memory to perform end-to-end event-based lip reading without any manual cropping.

Summary updated

Abstract

Event cameras offer high temporal resolution and low power consumption, making them promising for speech-related applications. These include automatic lip reading, audiovisual speech analysis, and human-robot interaction, particularly in challenging environments characterized by low illumination or high dynamic range. Event-based lip reading, speaker recognition, and voice activity detection, however, mostly operate on a cropped region of interest (ROI), by hand or by the dataset, and no event-based method supplies that ROI as a spatial region; existing event-based face analysis gives the mouth as landmarks, which presents a mismatch between application and localization. The present work predicts a dense heatmap over the lip region from the full event "frame". Events are encoded as a polarity-separated voxel grid and processed at every location by a two-pathway continuous-time recurrent network (CT-RNN) with learned, log-space bounded time constants acting on the real elapsed time of each bin: a stateless fast pathway and a stateful slow pathway, utilised for refining and anchoring the prediction. The model reaches 69.5% on lips on four unseen speakers and switches target only 2.1 times per second, against 8.9 and 11.0 for convolutional baselines with more parameters. Adding a spatial encoder improves precision but fails on a new face scale. On a real event sensor, trained only on simulated events, the proposed model reaches 58.0% on lips against 43.2% for the CNN, with fewer than half its target switches.
BibTeX
@misc{hammami2026event,
  author = {Hamze Hammami and Omar Eldardeer and Francesco Rea},
  title = {{Event-Based Dynamic Lip Region Estimation During Speech}},
  year = {2026},
  eprint = {2611.23111},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23111},
}
Figure