Learning Speech Representations by Predicting the Future

25 concepts25 papers1 proposal

When humans listen to speech, we do not process audio as an isolated sequence of independent pressure waves. Instead, our brains constantly anticipate what will be said next. If someone says "artificial intelligence and machine...", your brain instantly fills in the word "learning." This predictive capacity is not just a byproduct of understanding language; it is the very mechanism by which we learn to understand it in the first place.

In machine learning, translating this intuition into an unsupervised framework has led to a powerful class of self-supervised learning techniques known as predictive coding. By training neural networks to predict future or masked segments of an audio signal based solely on its past context, models can discover the underlying phonetic, syntactic, and semantic structures of speech without requiring a single line of human-annotated text.

The Core Intuition: Why Prediction Yields Structure

Raw audio is notoriously difficult for deep learning models to parse without supervision. Unlike text, which is neatly tokenized into discrete words or characters, speech is continuous, highly variable, and dense with low-level acoustic noise (such as background hums or room reverberations). If a model attempts to learn by simply memorizing the raw waveform, it wastes its capacity on these irrelevant details.

Predictive coding solves this by setting up a simple game: given a segment of audio up to time $t$, predict what the audio will look like at time $t + k$.

To succeed at this game over long time horizons, the model cannot rely on low-level acoustic continuity alone. It is forced to build abstract, high-level representations. For instance, to predict the next syllable or word boundary, the model must implicitly learn phonetic structures, speaker characteristics, and the statistical grammar of the language. As Ewan Dunbar et al. demonstrate in The Zero Resource Speech Challenge 2021: Spoken Language Modelling, this predictive mechanism allows systems to bootstrap acoustic, lexical, and semantic representations directly from raw audio waveforms without any textual transcriptions.

This predictive paradigm generally splits into two distinct strategies: Autoregressive Predictive Coding (APC), which focuses on generating the exact future signal, and Contrastive Predictive Coding (CPC), which focuses on distinguishing the true future from random alternatives.


Autoregressive Predictive Coding: Reconstructing the Future

Autoregressive Predictive Coding (APC) adopts a generative approach. It processes a sequence of acoustic features (like log-mel spectrogram frames) using a unidirectional network, such as a Long Short-Term Memory (LSTM) network or a Transformer, and attempts to reconstruct the exact spectral frame that occurs several steps ahead.

[ Past Frames ] ---> [ Context Encoder ] ---> [ Predicted Future Frame ]
                                                       |
                                            (Minimize L1/L2 Distance)
                                                       v
                                              [ Actual Future Frame ]

In Scaling Laws for Acoustic Models, Jasha Droppo et al. evaluate how this autoregressive framework scales across different architectures, establishing that predicting future speech frames serves as a robust foundation for acoustic modeling.

Because the model must output a concrete spectral frame, it learns fine-grained temporal structures and spectral patterns. This makes the learned features highly transferable to specialized acoustic tasks. For example:


Contrastive Predictive Coding: Choosing the Right Future

While APC works well, forcing a model to reconstruct exact high-dimensional frames can be inefficient. The model might expend too much energy trying to predict unpredictable background noise rather than the underlying message of the speech.

Contrastive Predictive Coding (CPC) bypasses this issue by turning the generation task into a multiple-choice discrimination task. Instead of predicting the exact future frame, the model projects the audio into a latent space and tries to select the true future latent representation out of a pool of random "distractor" segments.

                  +---> [ True Future Latent ]   --> (High Similarity)
                  |
[ Past Context ] -+---> [ Distractor Latent 1 ]  --> (Low Similarity)
                  |
                  +---> [ Distractor Latent 2 ]  --> (Low Similarity)

By optimizing a contrastive loss (such as InfoNCE), the model maximizes the mutual information between the present context and the future signal. As Alexei Baevski et al. show in their seminal work wav2vec: Unsupervised Pre-Training for Speech Recognition, this fully convolutional contrastive approach allows a network to learn high-quality speech representations from raw audio waveforms, drastically reducing the volume of labeled data needed to train state-of-the-art speech recognition systems.

Why the Contrastive Objective Works

By framing the problem contrastively, the network is discouraged from focusing on global noise that is shared across both the true future and the distractors. Instead, it must focus on the distinct phonetic and structural markers that uniquely tie the true future to the preceding context.

The authors of Unsupervised Pretraining Transfers Well Across Languages, Morgane Rivière et al., demonstrate that refining this contrastive objective to eliminate information leakage allows the model to capture universal features of human speech. Remarkably, a CPC model pre-trained entirely on English audio can seamlessly transfer its learned representations to boost speech recognition performance in completely unrelated, low-resource languages.

This same representational power extends beyond human speech. In Visualizing and sonifying how an artificial ear hears music, Vincent Herrmann applies CPC to audio scaleograms, revealing that predicting future musical segments naturally forces the network to learn abstract musical semantics—such as track structure and composer identity—without any explicit labels.


Scaling Up and Bridging the Domain Gap

Because predictive coding requires only raw audio, it can scale to massive, uncurated datasets. This capability is central to benchmarks like Libri-Light, introduced by Jacob Kahn et al. in Libri-Light: A Benchmark for ASR with Limited or No Supervision. By providing 60,000 hours of unlabelled speech, Libri-Light serves as a proving ground for tracking how effectively predictive coding models can discover phonetic structures directly from raw audio.

Furthermore, these learned representations have proven highly effective for downstream tasks that require separating overlapping signals or identifying health biomarkers:

Whether applied to target-domain data or transferred from out-of-domain sources, unsupervised predictive pre-training consistently provides a massive performance boost. As Ananya Misra et al. show in A Comparison of Supervised and Unsupervised Pre-Training of End-to-End Models, unsupervised pre-training on the target domain can yield Word Error Rate (WER) reductions of up to 30%, rivaling or outperforming supervised pre-training on alternative domains. This makes it an invaluable tool for deploying speech technology in real-world scenarios that lack clean data, such as building virtual assistants from noisy radio archives for low-resource languages (Moussa Doumbouya, Using Radio Archives for Low-Resource Speech Recognition: Towards an Intelligent Virtual Assistant for Illiterate Users).


Cognitive Connections and the "Ecological" Challenge

One of the most fascinating aspects of predictive coding is its alignment with biological intelligence. By training on raw audio without text, these models closely mimic how infants acquire their first language.

In Toward a realistic model of speech processing in the brain with self-supervised learning, Juliette MILLET et al. show that self-supervised predictive models (like wav2vec 2.0) trained on human-scale audio develop internal representations that closely mirror the speech processing hierarchy found in the human brain, as validated by fMRI recordings of people listening to audiobooks.

However, replicating true infant language acquisition requires moving past clean laboratory datasets. When models are trained on the messy, chaotic "ecological" audio that infants actually encounter in daily life, standard predictive coding frameworks face significant hurdles.

As explored by Maureen de Seyssel et al. in Realistic and broad-scope learning simulations: first results and challenges, statistical predictive learning can bootstrap phonetic and lexical properties simultaneously, but real-world environmental noise remains a massive obstacle. To bridge this gap, the authors of Modeling early phonetic acquisition from child-centered audio data, Marvin Lavechin et al., demonstrate that predictive coding models must be equipped with human-like inductive biases—such as the ability to actively filter for speech and track speaker consistency—to successfully learn phonetic structures from real-world, uncurated environments.

Conclusion

Predictive coding works because it turns time into a supervisor. By forcing a network to anticipate what comes next—either by generating the exact future frame or by selecting it from a crowd of distractors—the model naturally discards transient noise and retains the persistent, structural essence of the audio. The result is a highly robust, flexible, and biologically plausible framework for understanding the complex, continuous world of sound.

Key Foundational Work

Libri-Light: A Benchmark for ASR with Limited or No Supervision. (2019) — contributed 1 concept elements to this direction.
Pronunciation Erroneous Tendency Detection with Language Adversarial Represent Learning (2020) — contributed 1 concept elements to this direction.
Unsupervised pretraining transfers well across languages. (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_contrastive_predictive_coding_533dcb66,
  title = {contrastive predictive coding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/contrastive-predictive-coding/rdr_533dcb6624434193a98634ebd07e661e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_contrastive_predictive_coding_533dcb66_v1,
  title = {contrastive predictive coding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/contrastive-predictive-coding/rdr_533dcb6624434193a98634ebd07e661e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake