Multi-modal and Multi-scale Spatial Environment Understanding for Immersive Visual Text-to-Speech

Rui Liu, Shuwei He, Yifan Hu, Haizhou Li

Abstract

Visual Text-to-Speech (VTTS) aims to synthesize speech that is acoustically consistent with a given visual environment. Existing methods often fail to capture fine-grained local materials and precise spatial geometry, leading to unrealistic reverberation. We propose $M^2SE-VTTS$, a framework that leverages multi-modal (RGB and Depth) and multi-scale features. By incorporating LLM-generated semantic descriptions and a Top-k region selection mechanism, our model effectively captures both global context and local acoustic properties. Experimental results on the SoundSpaces-Speech dataset demonstrate that $M^2SE-VTTS$ significantly outperforms state-of-the-art baselines in both perceptual quality and acoustic accuracy.

Introduction

The goal of Visual Text-to-Speech (VTTS) is to generate speech that sounds as if it were recorded within a specific environment depicted in an image. For applications like virtual reality (VR) or augmented reality (AR), it is crucial that a character's voice reflects the acoustic properties of their surroundings—for instance, sounding resonant in a large hall but muffled in a small, carpeted room.

The primary challenge lies in spatial environment understanding. Previous research has largely focused on global features derived from RGB images, which often overlook two critical factors:

  1. Local Materials: Different surfaces (e.g., wood, fabric, tile) reflect sound differently. Global features often fail to capture these local nuances.
  2. Depth and Geometry: Sound propagation is inherently physical. Accurate modeling of echoes and reverberation requires knowledge of room dimensions and the distance to various surfaces.

Methodology

We introduce $M^2SE-VTTS$, a multi-modal and multi-scale approach designed to address these limitations. The architecture is divided into four main stages: feature extraction, local spatial understanding, local-aware global spatial understanding, and speech generation.

Overview of the $M^2SE-VTTS$ architecture, illustrating the multi-modal feature extraction and the multi-scale spatial understanding modules.

Multi-modal and Multi-scale Features

The framework processes both RGB images and depth maps to capture texture and physical dimensions simultaneously. To enhance semantic understanding, we utilize Gemini Pro Vision to generate descriptive captions of the scene. These captions provide high-level context that helps the model identify acoustically significant objects.

Local and Global Understanding

A key innovation is the Top-k Region Selection mechanism. Instead of processing the entire image uniformly, the model identifies the most relevant visual patches—such as hard floors or nearby walls—that significantly influence sound reflection. By sharing indices between RGB and depth features, the model ensures that material properties and spatial distances are considered in tandem. These local details are then fused with global context through a series of attention layers, ensuring the synthesized audio is tailored to the specific speaker location within the environment.

Experiments and Results

The model was evaluated using the SoundSpaces-Speech dataset, which features audio simulated within highly accurate 3D scans of real-world environments. Testing was conducted on both "seen" environments (included in training) and "unseen" environments (entirely new layouts).

Performance comparison of $M^2SE-VTTS$ against state-of-the-art baselines on the SoundSpaces-Speech dataset.

As shown in the results table, $M^2SE-VTTS$ consistently outperforms advanced baselines like VoiceLDM and DiffSpeech across all key metrics:

  • Mean Opinion Score (MOS): Achieved a score of 3.849 in unseen environments, indicating superior perceptual quality.
  • Reverberation Time Error (RTE): Demonstrated lower error in predicting reverberation characteristics, leading to more physically accurate echoes.
  • Mel Cepstral Distortion (MCD): Achieved the lowest distortion, signifying high spectral similarity to ground-truth audio.

Conclusion

$M^2SE-VTTS$ represents a significant step forward in immersive VTTS by integrating depth information and LLM-guided semantic understanding. By focusing on both local materials and global geometry, the model produces speech that is remarkably faithful to the visual environment. Future work will focus on reducing computational complexity for real-time applications and improving generalization across even more diverse architectural styles.

Summary updated

BibTeX
@inproceedings{liu2025multi,
  author = {Rui Liu and Shuwei He and Yifan Hu and Haizhou Li},
  title = {{Multi-modal and Multi-scale Spatial Environment Understanding for Immersive Visual Text-to-Speech}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i23.34643},
}
Figure