TL;DR
This paper introduces EgoSonics, a novel method for generating semantically meaningful and temporally synchronized audio for silent egocentric videos. By leveraging latent diffusion models and a new "SyncroNet" module, EgoSonics bridges the gap in current V2A synthesis, enabling richer applications in VR, assistive tech, and beyond.
What Problem Does This Paper Address?
While we've seen impressive progress in generating realistic videos from text, creating synchronized audio for silent videos remains a significant challenge. Existing methods for video-to-audio (V2A) synthesis are often limited to specific domains like speech or music, struggle with temporal synchronization, and operate on short video clips with restricted audio frequency ranges. This limitation hinders applications that rely on rich, contextually accurate soundscapes, such as immersive virtual reality experiences, assistive technologies for the visually impaired, or even enhancing existing video datasets. Egocentric videos, in particular, present a unique challenge due to their first-person perspective and the wide variety of everyday sounds they might contain.
Methods
EgoSonics tackles this problem by building upon the power of latent diffusion models (LDMs), specifically adapting Stable Diffusion. The core innovation lies in how they condition the audio generation process.
- Audio Representation: Instead of directly generating audio waveforms, EgoSonics converts raw audio into Short-Time Fourier Transform (STFT) "images" (spectrograms). This allows them to treat audio generation as an image generation task, leveraging the strengths of diffusion models.
- Video Encoding: Egocentric video frames are encoded using a CLIP image encoder to capture semantically rich information. This results in a video embedding that represents the temporal sequence of visual cues.
- SyncroNet Module: This is the key novel component. SyncroNet is built upon ControlNet and takes the video embedding as input. It generates "control signals" that provide pixel-level temporal guidance to the diffusion model. This module uses a combination of self-attention and cross-attention mechanisms to effectively inject time-aware information from the video into the audio generation process.
- Conditional Diffusion: The Stable Diffusion model is then conditioned on both the video embedding (for global context) and the control signals from SyncroNet (for local, temporal guidance). This allows for the generation of audio that is not only semantically relevant but also precisely synchronized with events in the video.
- Audio Upsampling: A learned audio upsampler is used to enhance the generated audio quality and recover higher frequencies.
Datasets
The primary dataset used for training and evaluation is Ego4D. This is a large-scale egocentric video dataset. The authors note that not all videos in Ego4D contain audio, and they focus on 10-second chunks with relevant visual audio cues. They also introduce a new method for evaluating audio-visual synchronization, the Video-Audio Alignment Score (VAAS), which uses a classifier trained on AV pairs to measure alignment.
Key Insights and Ideas
- Audio as Spectrograms: Treating audio spectrograms as images is a clever way to leverage powerful image generation models for audio synthesis.
- Temporal Control is Crucial: The success of EgoSonics hinges on the SyncroNet module's ability to extract and inject fine-grained temporal information from the video into the audio generation process. This is a significant step beyond methods that rely on average video embeddings.
- Egocentric Audio is Diverse: The paper highlights the need for models that can handle the broad range of sounds found in everyday egocentric videos, going beyond limited domains like speech or music.
- Implicit Noise Removal: The model appears to implicitly learn to suppress background noise, focusing on generating sounds directly related to visual events, which is a valuable emergent property.
Key Results
- Quantitative Performance: EgoSonics outperforms existing V2A methods (Im2Wav, Make-an-Audio, Diff-Foley) on key metrics like FID (Frechet Inception Distance), IS (Inception Score), and their proposed VAAS metric. For instance, their VAAS score reached 92.77 with 30 fps input, a significant improvement over baselines.
- Audio Quality: The generated audio is described as semantically meaningful, synchronized, and of higher quality, with a broader frequency support (up to 20 KHz).
- Video Summarization Improvement: When used to enhance video summarization models, EgoSonics' generated audio significantly improved the accuracy of predicting text embeddings, demonstrating its practical utility beyond just audio generation.
- User Study: A user study indicated that 80% of users found the generated audios realistic, and in over 75% of cases, users couldn't distinguish EgoSonics' output from ground truth audio.
Limitations and Future Work
The authors acknowledge several limitations:
- Occlusions and Lack of Visual Information: Misalignment can occur when visual information is occluded or insufficient to infer the corresponding sound.
- Data Scarcity: The model's performance is limited by the availability of diverse training data. For example, there are few samples of musical instruments in the Ego4D dataset, impacting performance on such videos.
Future work suggested includes training with larger datasets to address these limitations and further improving performance on challenging scenarios.
Summary updated