SpikeVoice: High-Quality Text-to-Speech Via Efficient Spiking Neural Network

Kexin Wang, Jiahong Zhang, Yong Ren, Man Yao, Di Shang, Bo Xu, Guoqi Li

Abstract

Brain-inspired Spiking Neural Networks (SNNs) have demonstrated effectiveness and efficiency in vision, natural language, and speech understanding tasks, indicating their capacity to "see," "listen," and "read." In this paper, we design SpikeVoice, which performs high-quality Text-To-Speech (TTS) via SNN, to explore the potential of SNNs to "speak." A major obstacle to using SNNs for such generative tasks lies in the demand for models to grasp long-term dependencies. The serial nature of spiking neurons leads to the invisibility of information at future spiking time steps, limiting SNN models to capturing sequence dependencies solely within the same time step—a phenomenon we term "partial-time dependency." To address this, we introduce Spiking Temporal-Sequential Attention (STSA) in SpikeVoice. To the best of our knowledge, SpikeVoice is the first TTS work in the SNN field. Experiments across four well-established datasets demonstrate that SpikeVoice achieves results comparable to Artificial Neural Networks (ANNs) with only 10.5% of the energy consumption.

Introduction

While traditional Artificial Neural Networks (ANNs) have achieved remarkable success in speech synthesis, their high energy consumption remains a significant barrier for deployment on edge devices. Spiking Neural Networks (SNNs) offer a bio-inspired alternative, utilizing discrete, binary "spikes" (0s and 1s) to communicate information. This sparse signaling allows SNNs to replace power-intensive Multiply-Accumulate (MAC) operations with simple additions. However, while SNNs have excelled in discriminative tasks like image classification, generative tasks like Text-to-Speech (TTS) have remained a challenge due to the inherent temporal limitations of spiking neurons.

The Challenge: Partial-Time Dependency

The primary obstacle in SNN-based TTS is what the researchers identify as partial-time dependency. In a standard SNN, neurons process information serially across time steps (charge, fire, and reset). Because of this sequential nature, a neuron at a given time step cannot "see" information from future time steps. In TTS, where the prosody and phonetics of a word often depend on the context provided by the entire sentence, this lack of global temporal context leads to poor synthesis quality and a failure to capture long-range dependencies.

The SpikeVoice Framework

SpikeVoice is a non-autoregressive TTS framework designed to overcome these limitations while maintaining the energy benefits of SNNs. It is built upon a Transformer-like architecture but is fully adapted for the spiking domain using Leaky Integrate-and-Fire (LIF) neurons.

Spiking Temporal-Sequential Attention (STSA)

To solve the partial-time dependency problem, SpikeVoice introduces the STSA module. This module operates in two distinct stages:

  1. Spiking Temporal Attention: This stage performs "temporal-mixing" across spiking time steps. By allowing the model to aggregate information across the temporal dimension before the neurons fire, it effectively grants the model the "global vision" required to understand sequence-wide context.
  2. Spiking Sequential Attention: This stage integrates contextual information across the phoneme sequence, ensuring that the relationships between different linguistic units are preserved.

By utilizing this dual-attention mechanism, SpikeVoice can capture complex dependencies that were previously inaccessible to standard SNN architectures.

Experimental Setup

The researchers evaluated SpikeVoice across four diverse datasets to test its robustness across languages and speaker configurations:

  • LJSpeech: English, single-speaker (24 hours).
  • Baker: Chinese, single-speaker (12 hours).
  • LibriTTS: English, multi-speaker.
  • AISHELL3: Chinese, multi-speaker (85 hours, 218 speakers).

Results and Energy Efficiency

SpikeVoice demonstrates that SNNs can compete with state-of-the-art ANNs in speech quality while operating at a fraction of the power cost.

  • Speech Quality: In Mean Opinion Score (MOS) evaluations, SpikeVoice achieved a score of 4.06 on the LJSpeech dataset, nearly matching the 4.10 score of the ANN-based FastSpeech2 baseline. Objective metrics like Word Error Rate (WER) and Character Error Rate (CER) also showed that SpikeVoice remains highly competitive with traditional models.
  • Energy Savings: The most striking result is the efficiency gain. SpikeVoice consumes only 10.5% of the energy required by an equivalent ANN. In configurations optimized for a single time step, this consumption can drop as low as 4.11%, representing a massive leap forward for green AI and on-device synthesis.

Discussion and Future Work

While SpikeVoice marks a milestone as the first high-quality SNN-based TTS system, several areas for improvement remain. The conversion of continuous text embeddings into binary spikes can lead to minor information loss, and the sequential nature of LIF neurons currently results in slower training times compared to parallelized ANNs. Additionally, the researchers observed that firing rates tend to increase in deeper layers; future iterations will focus on optimizing these rates to further enhance energy efficiency without compromising audio fidelity.

Conclusion

SpikeVoice proves that Spiking Neural Networks are capable of more than just "seeing" and "listening"—they can also "speak." By addressing the partial-time dependency bottleneck through STSA, SpikeVoice paves the way for a new generation of energy-efficient, high-quality generative speech models.

Summary updated

Abstract

Brain-inspired Spiking Neural Network (SNN) has demonstrated its effectiveness and efficiency in vision, natural language, and speech understanding tasks, indicating their capacity to "see", "listen", and "read". In this paper, we design \textbf{SpikeVoice}, which performs high-quality Text-To-Speech (TTS) via SNN, to explore the potential of SNN to "speak". A major obstacle to using SNN for such generative tasks lies in the demand for models to grasp long-term dependencies. The serial nature of spiking neurons, however, leads to the invisibility of information at future spiking time steps, limiting SNN models to capture sequence dependencies solely within the same time step. We term this phenomenon "partial-time dependency". To address this issue, we introduce Spiking Temporal-Sequential Attention STSA in the SpikeVoice. To the best of our knowledge, SpikeVoice is the first TTS work in the SNN field. We perform experiments using four well-established datasets that cover both Chinese and English languages, encompassing scenarios with both single-speaker and multi-speaker configurations. The results demonstrate that SpikeVoice can achieve results comparable to Artificial Neural Networks (ANN) with only 10.5 energy consumption of ANN.
BibTeX
@inproceedings{wang2024spikevoice,
  author = {Kexin Wang and Jiahong Zhang and Yong Ren and Man Yao and Di Shang and Bo Xu and Guoqi Li},
  title = {{SpikeVoice: High-Quality Text-to-Speech Via Efficient Spiking Neural Network}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=TBVTrxjWds},
}
Figure