SpikeVoice: High-Quality Text-to-Speech Via Efficient Spiking Neural Network

Kexin Wang, Jiahong Zhang, Yong Ren, Man Yao, Di Shang, Bo Xu, Guoqi Li

Abstract

Brain-inspired Spiking Neural Networks (SNNs) offer significant energy efficiency but have historically struggled with generative tasks like Text-to-Speech (TTS) due to difficulties in capturing long-term dependencies. We present SpikeVoice, the first high-quality TTS system built entirely on an SNN framework. By introducing Spiking Temporal-Sequential Attention (STSA), SpikeVoice overcomes the "partial-time dependency" limitation of traditional SNNs. Experimental results across multiple datasets show that SpikeVoice achieves speech quality comparable to Artificial Neural Networks (ANNs) while consuming only 10.5% of the energy.

Introduction

Spiking Neural Networks (SNNs) are energy-efficient, event-driven models that mimic biological neural processes. While they have succeeded in classification and recognition tasks, generative TTS remains challenging. The primary hurdle is partial-time dependency: because spiking neurons operate serially, they cannot access information from future time steps, hindering the capture of long-term sequence dependencies essential for natural speech.

Methodology

SpikeVoice is a Transformer-based SNN architecture designed to bridge the gap between efficiency and generative quality. It consists of a Spiking Phoneme Encoder (SPE), a Spiking Variance Adaptor (SVA) for pitch and duration, and a Spiking Mel Decoder (SMD).

Spiking Temporal-Sequential Attention (STSA)

The core innovation is the STSA module, which addresses the temporal bottleneck. It employs a two-stage mixing process:

  1. Temporal-Mixing: Integrates global information across the spiking time dimension from binary embeddings.
  2. Sequential-Mixing: Captures contextual information across the phoneme sequence.
The SpikeVoice architecture, highlighting the STSA module and the overall flow from phoneme embedding to mel-spectrogram generation.

Spike-Driven Paradigm

The model utilizes Leaky Integrate-and-Fire (LIF) neurons. By communicating via sparse binary spikes, the network replaces power-intensive Multiply-and-Accumulate (MAC) operations with efficient Accumulate (AC) operations, which are natively supported by neuromorphic hardware.

Experiments and Results

We evaluated SpikeVoice on four datasets: LJSpeech, Baker, LibriTTS, and AISHELL3, covering single and multi-speaker scenarios in English and Chinese.

Speech Quality and Accuracy

SpikeVoice achieves a Mean Opinion Score (MOS) of 4.06 on LJSpeech, nearly matching the ANN baseline (FastSpeech2) at 4.10. Furthermore, it often outperforms the baseline in Word Error Rate (WER) and Character Error Rate (CER), suggesting that SNN temporal dynamics may improve alignment and clarity.

Comparison of mel-spectrograms generated by FastSpeech2 and various SpikeVoice configurations. SpikeVoice-STSA shows high fidelity comparable to the ANN baseline.

Energy Efficiency

The most striking result is the energy reduction. SpikeVoice consumes only 10.5% of the energy of its ANN counterpart. Under specific configurations (time step = 1), this can drop as low as 4.11%, demonstrating the immense potential of SNNs for low-power edge devices.

Conclusion

SpikeVoice demonstrates that SNNs are capable of high-fidelity generative tasks. By solving the partial-time dependency problem with STSA, we achieve state-of-the-art TTS performance with a fraction of the energy cost of traditional ANNs. Future work will focus on reducing information loss in multi-speaker settings and improving training speeds through parallelization.

Summary updated

Abstract

Brain-inspired Spiking Neural Network (SNN) has demonstrated its effectiveness and efficiency in vision, natural language, and speech understanding tasks, indicating their capacity to “see”, “listen”, and “read”. In this paper, we design SpikeVoice, which performs high-quality Text-To-Speech (TTS) via SNN, to explore the potential of SNN to “speak”. A major obstacle to using SNN for such generative tasks lies in the demand for models to grasp long-term dependencies. The serial nature of spiking neurons, however, leads to the invisibility of information at future spiking time steps, limiting SNN models to capture sequence dependencies solely within the same time step. We term this phenomenon “partial-time dependency”. To address this issue, we introduce Spiking Temporal-Sequential Attention (STSA) in the SpikeVoice. To the best of our knowledge, SpikeVoice is the first TTS work in the SNN field. We perform experiments using four well-established datasets that cover both Chinese and English languages, encompassing scenarios with both single-speaker and multi-speaker configurations. The results demonstrate that SpikeVoice can achieve results comparable to Artificial Neural Networks (ANN) with only 10.5% energy consumption of ANN. Both our demo and code are available as supplementary material.
BibTeX
@inproceedings{wang2024spikevoice,
  author = {Kexin Wang and Jiahong Zhang and Yong Ren and Man Yao and Di Shang and Bo Xu and Guoqi Li},
  title = {{SpikeVoice: High-Quality Text-to-Speech Via Efficient Spiking Neural Network}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=TBVTrxjWds},
}
Figure