SIREN: Spatially-Informed Reconstruction of Binaural Audio with Vision

Mingyeong Song, Seoyeon Ko, Junhyug Noh

TL;DR

This paper introduces SIREN, a novel framework that reconstructs immersive binaural audio from monaural video by explicitly predicting left and right audio channels. It leverages a Vision Transformer (ViT) with dual-head attention for spatially-informed audio generation and a confidence-weighted refinement strategy to suppress artifacts, achieving state-of-the-art results on audio-visual binaural reconstruction benchmarks.

What Problem Does This Paper Address?

Many consumer videos are recorded with monaural audio, lacking the spatial cues essential for immersive experiences like VR, gaming, and interactive media. While existing methods attempt to convert monaural audio to binaural using visual information, they face challenges: conventional visual models might miss crucial spatial details, the guidance for left/right channel generation is often heuristic rather than learned end-to-end, and aggregating predictions from different video segments can lead to audio leakage and unstable spatialization. SIREN aims to overcome these limitations by providing a more robust and accurate method for generating binaural audio from monaural sources guided by video.

Methods

SIREN employs a multi-faceted approach:

  • Vision Transformer (ViT) with Dual-Head Self-Attention: Instead of relying on hand-crafted masks, SIREN uses a ViT encoder with dual-head self-attention. This allows the model to learn explicit left and right directional cues directly from the visual input. The attention mechanism produces a shared scene map and separate left/right attention maps, which are then used to modulate the audio generation process.
  • Soft Spatial Prior: To stabilize the initial learning of directional cues, a soft spatial prior is introduced. This prior, implemented as decaying logistic target maps, provides an initial bias towards left/right grounding early in training and gradually fades away, allowing the model to learn content-driven spatialization.
  • Audio U-Net with FiLM Conditioning: An audio U-Net architecture is used for generating the binaural audio. It takes the monaural audio spectrogram and is conditioned by visual features from the ViT using FiLM (Feature-wise Linear Modulation) layers. This allows the visual context to dynamically influence the audio generation at different resolutions.
  • Confidence-Weighted Test-Time Refinement: At inference, to handle potential artifacts from aggregating predictions across multiple video crops and overlapping segments, SIREN uses a two-stage confidence-weighted refinement. This process leverages two key scores:
  • Mono Consistency: Measures how well the predicted binaural audio's mono mix matches the original monaural input, ensuring spectral fidelity.
  • Interaural Phase Consistency: Evaluates the agreement of interaural phase cues in the reconstructed audio, crucial for spatial perception. These scores are used to weight and fuse predictions, effectively suppressing crosstalk and improving spatial stability.

Datasets

SIREN is evaluated on two widely used audio-visual benchmarks:

  • FAIR-Play: This dataset consists of professional binaural microphone recordings with synchronized video. The paper uses the official 10-split protocol and also refers to a more challenging 5-split protocol.
  • MUSIC-Stereo: This dataset is derived from MUSIC, focusing on videos with clear interaural differences. The authors filtered the dataset to ensure distinct spatial cues and used a standard train/val/test split.

Key Insights and Ideas

  • Transformer-Native Directional Cues: The use of dual-head self-attention within the ViT for explicit L/R attention maps is a significant departure from previous methods. This allows for end-to-end, data-driven learning of directional cues, replacing rigid heuristics.
  • Annealing Spatial Prior for Robust Grounding: The soft, annealed spatial prior is an elegant way to guide the model's initial learning without imposing overly strict constraints, allowing for flexibility as training progresses.
  • Confidence-Weighted Fusion for Artifact Reduction: The two-stage confidence-weighted refinement is a clever post-processing step that significantly improves the robustness of binaural audio generation, particularly when dealing with aggregated predictions from multiple sources. It directly addresses the issue of L/R leakage.
  • Synergy of Visual and Audio Modalities: The paper demonstrates how fine-grained visual information, processed by a powerful ViT, can be effectively translated into spatial audio cues, highlighting the growing importance of audio-visual learning.

Key Results

SIREN achieves state-of-the-art performance on both FAIR-Play and MUSIC-Stereo datasets across various metrics:

  • On MUSIC-Stereo, SIREN achieves the lowest STFT, Envelope Distance, and Phase Distance scores, indicating superior time-frequency fidelity, perceptual similarity, and phase accuracy.
  • On FAIR-Play, SIREN achieves the best STFT and Signal-to-Noise Ratio (SNR), with a competitive Phase Distance score.
  • Ablation studies show that both the spatial prior loss and the confidence-weighted refinement contribute significantly to the overall performance, with the refinement being particularly effective in improving fidelity and reducing artifacts.
  • The dual-head attention and FiLM conditioning are shown to be crucial for effective spatial cue extraction and audio generation, respectively.

Limitations and Future Work

  • Phase Distance on FAIR-Play: The authors note a modest gap in Phase Distance on the FAIR-Play dataset compared to CMC. They attribute this to the explicit L/R prediction and waveform-domain fusion, which might introduce minor interaural phase mismatches in controlled recordings. They suggest that increasing the phase weight or incorporating IPD/group-delay terms could mitigate this.
  • Future Directions: While not explicitly detailed, the modular design of SIREN suggests it could be extended to other audio-visual tasks or adapted for real-time applications. The success of transformer-native attention for directional cues also opens avenues for exploring more complex spatial audio phenomena.

Summary updated

Abstract

Binaural audio delivers spatial cues essential for immersion, yet most consumer videos are monaural due to capture constraints. We introduce SIREN, a visually guided mono to binaural framework that explicitly predicts left and right channels. A ViT-based encoder learns dual-head self-attention to produce a shared scene map and end-to-end L/R attention, replacing hand-crafted masks. A soft, annealed spatial prior gently biases early L/R grounding, and a two-stage, confidence-weighted waveform-domain fusion (guided by mono reconstruction and interaural phase consistency) suppresses crosstalk when aggregating multi-crop and overlapping windows. Evaluated on FAIR-Play and MUSIC-Stereo, SIREN yields consistent gains on time-frequency and phase-sensitive metrics with competitive SNR. The design is modular and generic, requires no task-specific annotations, and integrates with standard audio-visual pipelines.
BibTeX
@misc{song2026siren,
  author = {Mingyeong Song and Seoyeon Ko and Junhyug Noh},
  title = {{SIREN: Spatially-Informed Reconstruction of Binaural Audio with Vision}},
  year = {2026},
  eprint = {2603.29820},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.29820},
}
Figure