Abstract
Visual Text-to-Speech (VTTS) aims to generate speech that reflects the acoustic characteristics of a specific visual environment. Traditional methods often rely on global RGB features, neglecting local material properties and spatial geometry. We introduce M²SE-VTTS, a novel framework that integrates RGB and Depth data with semantic guidance from Large Language Models (LLMs). Our approach employs a multi-scale understanding scheme to capture fine-grained acoustic cues. Experimental results on the SoundSpaces-Speech dataset demonstrate that M²SE-VTTS significantly improves speech naturalness and acoustic consistency, outperforming current state-of-the-art models in both seen and unseen environments.
Introduction
The goal of Visual Text-to-Speech (VTTS) is to synthesize speech that sounds as if it were recorded within a specific visual scene. For instance, in a virtual reality application, a character's voice should realistically transition from the muffled tones of a carpeted bedroom to the sharp echoes of a tiled bathroom.
The primary challenge lies in accurately inferring the acoustic properties of a space from visual inputs. Previous research has predominantly focused on global visual features from RGB images. However, acoustics are fundamentally determined by two factors often overlooked: local material properties (e.g., wood vs. plush fabric) and spatial geometry (e.g., room dimensions and speaker-microphone distance). Without accounting for these elements, synthesized speech often lacks the immersive quality required for high-fidelity applications.
Methodology
The proposed Multi-modal and Multi-scale Spatial Environment (M²SE) understanding scheme addresses these challenges through a four-stage pipeline.
- Multi-modal Feature Extraction: The system processes both an RGB image and a Depth map using a pre-trained CLIP-ViT encoder. This allows for the extraction of both patch-level (local) and global features, capturing texture and depth simultaneously.
- Local Spatial Understanding: To prioritize acoustically significant regions, the model utilizes Gemini Pro Vision to generate descriptive scene captions. This semantic information guides a Top-k Regions Detector to identify areas likely to influence sound reflection and absorption.
- Local-aware Global Spatial Understanding: A "shared index" strategy is employed to ensure spatial coherence. By using the same patch indices for both RGB and Depth modalities, the model correlates material properties with their specific spatial locations.
- Speech Generation: The enriched features are integrated into a TTS backbone. A variance adaptor and denoiser refine the audio representations, which are then converted into high-fidelity waveforms using the BigVGAN vocoder.
Experimental Results
The model was trained and evaluated on the SoundSpaces-Speech dataset, which utilizes real-world 3D scans to simulate accurate environmental audio. The evaluation included both "Test-Seen" (known environments) and "Test-Unseen" (novel environments) splits.
As shown in the results, M²SE-VTTS consistently outperforms advanced baselines, including diffusion-based models like ProDiff and VoiceLDM. Key performance indicators include:
- Mean Opinion Score (MOS): Achieved a score of 3.849 on unseen environments, surpassing the previous state-of-the-art.
- Reverberation Time Error (RTE): Demonstrated the lowest error (0.0744), indicating superior understanding of room echoes.
- Mel Cepstral Distortion (MCD): Achieved the lowest distortion (4.4215), reflecting high spectral similarity to ground truth recordings.
Discussion and Conclusion
A critical finding of this work is the effectiveness of the "shared index" strategy. Correlating the material (RGB) and distance (Depth) of the same object proves vital for accurate acoustic modeling. Furthermore, the integration of LLM-generated semantic captions provides a powerful mechanism for focusing the model's attention on relevant visual cues.
While M²SE-VTTS sets a new benchmark, future work will focus on improving computational efficiency for real-time deployment in mobile AR/VR settings and enhancing robustness in dynamic scenes with moving objects.
Summary updated