Abstract
Brain-inspired Spiking Neural Networks (SNNs) offer significant energy efficiency but have historically struggled with generative tasks like Text-to-Speech (TTS) due to difficulties in capturing long-term dependencies. We present SpikeVoice, the first high-quality TTS system built entirely on an SNN framework. By introducing Spiking Temporal-Sequential Attention (STSA), SpikeVoice overcomes the "partial-time dependency" limitation of traditional SNNs. Experimental results across multiple datasets show that SpikeVoice achieves speech quality comparable to Artificial Neural Networks (ANNs) while consuming only 10.5% of the energy.
Introduction
Spiking Neural Networks (SNNs) are energy-efficient, event-driven models that mimic biological neural processes. While they have succeeded in classification and recognition tasks, generative TTS remains challenging. The primary hurdle is partial-time dependency: because spiking neurons operate serially, they cannot access information from future time steps, hindering the capture of long-term sequence dependencies essential for natural speech.
Methodology
SpikeVoice is a Transformer-based SNN architecture designed to bridge the gap between efficiency and generative quality. It consists of a Spiking Phoneme Encoder (SPE), a Spiking Variance Adaptor (SVA) for pitch and duration, and a Spiking Mel Decoder (SMD).
Spiking Temporal-Sequential Attention (STSA)
The core innovation is the STSA module, which addresses the temporal bottleneck. It employs a two-stage mixing process:
- Temporal-Mixing: Integrates global information across the spiking time dimension from binary embeddings.
- Sequential-Mixing: Captures contextual information across the phoneme sequence.
Spike-Driven Paradigm
The model utilizes Leaky Integrate-and-Fire (LIF) neurons. By communicating via sparse binary spikes, the network replaces power-intensive Multiply-and-Accumulate (MAC) operations with efficient Accumulate (AC) operations, which are natively supported by neuromorphic hardware.
Experiments and Results
We evaluated SpikeVoice on four datasets: LJSpeech, Baker, LibriTTS, and AISHELL3, covering single and multi-speaker scenarios in English and Chinese.
Speech Quality and Accuracy
SpikeVoice achieves a Mean Opinion Score (MOS) of 4.06 on LJSpeech, nearly matching the ANN baseline (FastSpeech2) at 4.10. Furthermore, it often outperforms the baseline in Word Error Rate (WER) and Character Error Rate (CER), suggesting that SNN temporal dynamics may improve alignment and clarity.
Energy Efficiency
The most striking result is the energy reduction. SpikeVoice consumes only 10.5% of the energy of its ANN counterpart. Under specific configurations (time step = 1), this can drop as low as 4.11%, demonstrating the immense potential of SNNs for low-power edge devices.
Conclusion
SpikeVoice demonstrates that SNNs are capable of high-fidelity generative tasks. By solving the partial-time dependency problem with STSA, we achieve state-of-the-art TTS performance with a fraction of the energy cost of traditional ANNs. Future work will focus on reducing information loss in multi-speaker settings and improving training speeds through parallelization.
Summary updated