EvMic: Event-based Non-contact sound recovery from effective spatial-temporal modeling

Hao Yin, Shi Guo, Xu Jia, Xudong Xu, Lu Zhang, Si Liu, Dong Wang, Huchuan Lu, Tianfan Xue

Abstract

Non-contact sound recovery, or the "visual microphone," aims to reconstruct audio from microscopic object vibrations. Traditional high-speed cameras are limited by a bandwidth bottleneck, while event cameras offer high temporal resolution with low data overhead. This paper introduces EvMic, a deep-learning pipeline that utilizes laser-assisted illumination and a novel neural architecture to recover high-fidelity audio. EvMic employs sparse convolutions for efficient feature extraction and Mamba-based state-space models for long-range temporal modeling. Our approach achieves state-of-the-art performance, significantly outperforming existing frame-based and event-based methods in both signal quality and speech intelligibility.

Introduction: The Bandwidth Bottleneck

The concept of recovering sound by observing microscopic vibrations—such as those on a chip bag or speaker cone—has long been a goal in computational imaging. However, traditional high-speed cameras face a severe "bandwidth bottleneck": capturing the high-frequency vibrations necessary for audio requires extremely high frame rates, generating massive amounts of data that limit the field of view or sampling duration.

Comparison between traditional frame-based and the proposed event-based sound recovery.

Event cameras offer a promising alternative by recording pixel-level brightness changes asynchronously. This provides microsecond-level temporal resolution with minimal data redundancy. EvMic addresses the challenge of noisy reconstruction in existing event-based methods by fully exploiting spatial-temporal context through a specialized deep-learning architecture.

Methods: A Three-Stage Neural Architecture

EvMic's pipeline is designed to handle the unique sparsity and high temporal density of event data through four key components:

  1. Laser-Assisted Imaging: To amplify subtle vibrations, a laser matrix projects "speckles" onto the target object. These high-gradient points ensure that even minute movements trigger a dense stream of events.
  2. Lightweight Feature Extraction (SPconv): Utilizing the inherent sparsity of event data, the model employs Sparse Convolutions (ResNet18-based). This allows the system to process only active pixels, reducing memory usage to ~33% and FLOPs to ~4% compared to traditional CNNs.
  3. Spatial Aggregation Block (SAB): Since vibration visibility depends on surface geometry, the SAB uses multi-head self-attention to adaptively aggregate information from different laser speckles, filtering interference and boosting the signal-to-noise ratio.
  4. Temporal Modeling (Mamba): High-fidelity audio recovery requires modeling long sequences (4,000 bins per second). EvMic uses Mamba, a state-space model (SSM) that captures long-range dependencies with linear scaling, offering superior efficiency over standard Transformers.
The overall framework of EvMic, featuring sparse backbone, spatial aggregation, and Mamba-based temporal modeling.

Key Results and Insights

EvMic was evaluated on a newly created synthetic dataset (simulated via Blender and v2e) and real-world captures using a Prophesee EVK4 event camera.

  • Quantitative Superiority: EvMic achieved an average Signal-to-Noise Ratio (SNR) of 1.214 dB, significantly outperforming the previous state-of-the-art event-based method, EvPhase (-0.079 dB), and frame-based RGBPhase (-2.837 dB).
  • Speech Intelligibility: The model showed a marked improvement in the STOI metric, reaching 0.481 compared to 0.376 for previous methods.
  • Advanced Capabilities: The system successfully demonstrated stereo audio recovery from two simultaneous sound sources and was sensitive enough to visualize physical phase shifts of sound waves propagating across objects.
  • Efficiency: By treating sparsity as a feature, the sparse backbone drastically reduced computational overhead without sacrificing performance.

Conclusion

EvMic represents a significant step forward in non-contact sound recovery. By combining event-based sensing with efficient sparse convolutions and Mamba-based temporal modeling, it overcomes traditional hardware limitations. While challenges remain in narrowing the sim-to-real gap and handling environmental sensitivity, EvMic establishes a new state-of-the-art for visual microphones.

Summary updated

Abstract

When sound waves hit an object, they induce vibrations that produce high-frequency and subtle visual changes, which can be used for recovering the sound. Early studies always encounter trade-offs related to sampling rate, bandwidth, field of view, and the simplicity of the optical path. Recent advances in event camera hardware show good potential for its application in visual sound recovery, because of its superior ability in capturing high-frequency signals. However, existing event-based vibration recovery methods are still sub-optimal for sound recovery. In this work, we propose a novel pipeline for non-contact sound recovery, fully utilizing spatial-temporal information from the event stream. We first generate a large training set using a novel simulation pipeline. Then we designed a network that leverages the sparsity of events to capture spatial information and uses Mamba to model long-term temporal information. Lastly, we train a spatial aggregation block to aggregate information from different locations to further improve signal quality. To capture event signals caused by sound waves, we also designed an imaging system using a laser matrix to enhance the gradient and collected multiple data sequences for testing. Experimental results on synthetic and real-world data demonstrate the effectiveness of our method.
BibTeX
@misc{yin2025evmic,
  author = {Hao Yin and Shi Guo and Xu Jia and Xudong XU and Lu Zhang and Si Liu and Dong Wang and Huchuan Lu and Tianfan Xue},
  title = {{EvMic: Event-based Non-contact sound recovery from effective spatial-temporal modeling}},
  year = {2025},
  eprint = {2504.02402},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.02402},
}
Figure