Multi-modal and Multi-scale Spatial Environment Understanding for Immersive Visual Text-to-Speech

Rui Liu, Shuwei He, Yifan Hu, Haizhou Li

Abstract

Visual Text-to-Speech (VTTS) aims to generate speech that reflects the acoustic characteristics of a specific visual environment. Traditional methods often rely on global RGB features, neglecting local material properties and spatial geometry. We introduce M²SE-VTTS, a novel framework that integrates RGB and Depth data with semantic guidance from Large Language Models (LLMs). Our approach employs a multi-scale understanding scheme to capture fine-grained acoustic cues. Experimental results on the SoundSpaces-Speech dataset demonstrate that M²SE-VTTS significantly improves speech naturalness and acoustic consistency, outperforming current state-of-the-art models in both seen and unseen environments.

Introduction

The goal of Visual Text-to-Speech (VTTS) is to synthesize speech that sounds as if it were recorded within a specific visual scene. For instance, in a virtual reality application, a character's voice should realistically transition from the muffled tones of a carpeted bedroom to the sharp echoes of a tiled bathroom.

The primary challenge lies in accurately inferring the acoustic properties of a space from visual inputs. Previous research has predominantly focused on global visual features from RGB images. However, acoustics are fundamentally determined by two factors often overlooked: local material properties (e.g., wood vs. plush fabric) and spatial geometry (e.g., room dimensions and speaker-microphone distance). Without accounting for these elements, synthesized speech often lacks the immersive quality required for high-fidelity applications.

Methodology

The proposed Multi-modal and Multi-scale Spatial Environment (M²SE) understanding scheme addresses these challenges through a four-stage pipeline.

M²SE-VTTS Architecture Overview
  1. Multi-modal Feature Extraction: The system processes both an RGB image and a Depth map using a pre-trained CLIP-ViT encoder. This allows for the extraction of both patch-level (local) and global features, capturing texture and depth simultaneously.
  2. Local Spatial Understanding: To prioritize acoustically significant regions, the model utilizes Gemini Pro Vision to generate descriptive scene captions. This semantic information guides a Top-k Regions Detector to identify areas likely to influence sound reflection and absorption.
  3. Local-aware Global Spatial Understanding: A "shared index" strategy is employed to ensure spatial coherence. By using the same patch indices for both RGB and Depth modalities, the model correlates material properties with their specific spatial locations.
  4. Speech Generation: The enriched features are integrated into a TTS backbone. A variance adaptor and denoiser refine the audio representations, which are then converted into high-fidelity waveforms using the BigVGAN vocoder.

Experimental Results

The model was trained and evaluated on the SoundSpaces-Speech dataset, which utilizes real-world 3D scans to simulate accurate environmental audio. The evaluation included both "Test-Seen" (known environments) and "Test-Unseen" (novel environments) splits.

Performance comparison against state-of-the-art baselines

As shown in the results, M²SE-VTTS consistently outperforms advanced baselines, including diffusion-based models like ProDiff and VoiceLDM. Key performance indicators include:

  • Mean Opinion Score (MOS): Achieved a score of 3.849 on unseen environments, surpassing the previous state-of-the-art.
  • Reverberation Time Error (RTE): Demonstrated the lowest error (0.0744), indicating superior understanding of room echoes.
  • Mel Cepstral Distortion (MCD): Achieved the lowest distortion (4.4215), reflecting high spectral similarity to ground truth recordings.

Discussion and Conclusion

A critical finding of this work is the effectiveness of the "shared index" strategy. Correlating the material (RGB) and distance (Depth) of the same object proves vital for accurate acoustic modeling. Furthermore, the integration of LLM-generated semantic captions provides a powerful mechanism for focusing the model's attention on relevant visual cues.

While M²SE-VTTS sets a new benchmark, future work will focus on improving computational efficiency for real-time deployment in mobile AR/VR settings and enhancing robustness in dynamic scenes with moving objects.

Summary updated

Abstract

Visual Text-to-Speech (VTTS) aims to take the environmental image as the prompt to synthesize the reverberant speech for the spoken content. The challenge of this task lies in understanding the spatial environment from the image. Many attempts have been made to extract global spatial visual information from the RGB space of an spatial image. However, local and depth image information are crucial for understanding the spatial environment, which previous works have ignored. To address the issues, we propose a novel multi-modal and multi-scale spatial environment understanding scheme to achieve immersive VTTS, termed M2SE-VTTS. The multi-modal aims to take both the RGB and Depth spaces of the spatial image to learn more comprehensive spatial information, and the multi-scale seeks to model the local and global spatial knowledge simultaneously. Specifically, we first split the RGB and Depth images into patches and adopt the Gemini-generated environment captions to guide the local spatial understanding. After that, the multi-modal and multi-scale features are integrated by the local-aware global spatial understanding. In this way, M2SE-VTTS effectively models the interactions between local and global spatial contexts in the multi-modal spatial environment. Objective and subjective evaluations suggest that our model outperforms the advanced baselines in environmental speech generation.
BibTeX
@inproceedings{liu2025multi,
  author = {Rui Liu and Shuwei He and Yifan Hu and Haizhou Li},
  title = {{Multi-modal and Multi-scale Spatial Environment Understanding for Immersive Visual Text-to-Speech}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i23.34643},
}
Figure