Abstract
In this work, we propose a new recurrent autoencoder architecture, termed Feedback Recurrent AutoEncoder (FRAE), for online compression of sequential data with temporal dependency. The recurrent structure of FRAE is designed to efficiently extract the redundancy along the time dimension and allows a compact discrete representation of the data to be learned. We demonstrate its effectiveness in speech spectrogram compression. Specifically, we show that the FRAE, paired with a powerful neural vocoder, can produce high-quality speech waveforms at a low, fixed bitrate. We further show that by adding a learned prior for the latent space and using an entropy coder, we can achieve an even lower variable bitrate.
Introduction
Compressing sequential data like speech or video is notoriously difficult because of temporal dependencies—what happens now is highly correlated with what happened a moment ago. Standard autoencoders (AEs) often treat data points independently or use "forward-only" architectures that fail to capture long-range context in an online (real-time) fashion.
Existing recurrent models often suffer from a "mismatch": the encoder might have access to the original high-fidelity history, but the decoder only sees a lossy, quantized version. This leads to inefficiencies where the encoder sends redundant information that the decoder could have predicted on its own. The challenge is to create a system where the encoder knows exactly what the decoder's current state is, allowing it to send only the most essential "corrections."
The Feedback Recurrent AutoEncoder (FRAE)
The authors introduce the Feedback Recurrent AutoEncoder (FRAE). Its defining feature is a feedback loop where the decoder’s recurrent state ($h_t$) is fed back to the encoder at the next time step. This allows the system to perform "analysis-by-synthesis": because the encoder receives the decoder's state, it knows what the decoder is likely to predict and only encodes the residual information (the difference between the prediction and reality).
To achieve compression, the latent representation ($z_t$) is passed through a discrete bottleneck using a learned codebook. Research showed that feeding back the hidden state ($h_t$) is significantly more effective and stable than feeding back the reconstructed output ($\hat{x}_t$), which can lead to training instability.
Variational Extension and Prior Learning
The FRAE can be extended into a Variational AutoEncoder (FR-VAE) by adding a trainable probability model (a prior). By making the prior distribution of the latent space dependent on the decoder's state ($p(z_t | h_{t-1})$), the model learns to anticipate which codes are most likely. This allows for variable bitrate encoding using entropy coding, further squeezing the data by approximately 1.5 Kbps without losing quality.
Experimental Results
The models were evaluated on the LibriVox and WSJ1 speech datasets. At a fixed bitrate of 1.6 Kbps, FRAE achieved a Mel-scale MSE of 13.003, significantly lower than "Encoder-only" (18.343) or "Separate" recurrent (14.283) architectures.
When paired with a WaveNet vocoder for final synthesis, the FRAE system achieved higher POLQA scores (an objective measure of speech quality) than the industry-standard Opus codec at comparable bitrates.
As shown in the figure, the gap in quality is especially pronounced at the lowest bitrates (2–6 Kbps), where FRAE maintains high perceptual quality while Opus performance declines.
Conclusion
The FRAE architecture provides a smarter way to compress sequential data by leveraging decoder state feedback to eliminate temporal redundancy. By evolving classical predictive coding into a deep learning framework, it achieves state-of-the-art results in speech compression. Future work may incorporate multi-scale latents to capture correlations at different time scales and tighter integration of phase information to further improve naturalness.
Summary updated