Abstract
Non-contact sound recovery, or the "visual microphone," aims to reconstruct audio from microscopic object vibrations. Traditional high-speed cameras are limited by a bandwidth bottleneck, while event cameras offer high temporal resolution with low data overhead. This paper introduces EvMic, a deep-learning pipeline that utilizes laser-assisted illumination and a novel neural architecture to recover high-fidelity audio. EvMic employs sparse convolutions for efficient feature extraction and Mamba-based state-space models for long-range temporal modeling. Our approach achieves state-of-the-art performance, significantly outperforming existing frame-based and event-based methods in both signal quality and speech intelligibility.
Introduction: The Bandwidth Bottleneck
The concept of recovering sound by observing microscopic vibrations—such as those on a chip bag or speaker cone—has long been a goal in computational imaging. However, traditional high-speed cameras face a severe "bandwidth bottleneck": capturing the high-frequency vibrations necessary for audio requires extremely high frame rates, generating massive amounts of data that limit the field of view or sampling duration.
Event cameras offer a promising alternative by recording pixel-level brightness changes asynchronously. This provides microsecond-level temporal resolution with minimal data redundancy. EvMic addresses the challenge of noisy reconstruction in existing event-based methods by fully exploiting spatial-temporal context through a specialized deep-learning architecture.
Methods: A Three-Stage Neural Architecture
EvMic's pipeline is designed to handle the unique sparsity and high temporal density of event data through four key components:
- Laser-Assisted Imaging: To amplify subtle vibrations, a laser matrix projects "speckles" onto the target object. These high-gradient points ensure that even minute movements trigger a dense stream of events.
- Lightweight Feature Extraction (SPconv): Utilizing the inherent sparsity of event data, the model employs Sparse Convolutions (ResNet18-based). This allows the system to process only active pixels, reducing memory usage to ~33% and FLOPs to ~4% compared to traditional CNNs.
- Spatial Aggregation Block (SAB): Since vibration visibility depends on surface geometry, the SAB uses multi-head self-attention to adaptively aggregate information from different laser speckles, filtering interference and boosting the signal-to-noise ratio.
- Temporal Modeling (Mamba): High-fidelity audio recovery requires modeling long sequences (4,000 bins per second). EvMic uses Mamba, a state-space model (SSM) that captures long-range dependencies with linear scaling, offering superior efficiency over standard Transformers.
Key Results and Insights
EvMic was evaluated on a newly created synthetic dataset (simulated via Blender and v2e) and real-world captures using a Prophesee EVK4 event camera.
- Quantitative Superiority: EvMic achieved an average Signal-to-Noise Ratio (SNR) of 1.214 dB, significantly outperforming the previous state-of-the-art event-based method, EvPhase (-0.079 dB), and frame-based RGBPhase (-2.837 dB).
- Speech Intelligibility: The model showed a marked improvement in the STOI metric, reaching 0.481 compared to 0.376 for previous methods.
- Advanced Capabilities: The system successfully demonstrated stereo audio recovery from two simultaneous sound sources and was sensitive enough to visualize physical phase shifts of sound waves propagating across objects.
- Efficiency: By treating sparsity as a feature, the sparse backbone drastically reduced computational overhead without sacrificing performance.
Conclusion
EvMic represents a significant step forward in non-contact sound recovery. By combining event-based sensing with efficient sparse convolutions and Mamba-based temporal modeling, it overcomes traditional hardware limitations. While challenges remain in narrowing the sim-to-real gap and handling environmental sensitivity, EvMic establishes a new state-of-the-art for visual microphones.
Summary updated