Intermediate Speech Representations for Audio Deepfake Detection

19 concepts11 papers1 proposal

When deep neural networks are trained on speech, they are typically incentivized to optimize for semantic invariance. To successfully transcribe an audio clip or recognize a spoken command, a model must understand the underlying words regardless of background noise, microphone quality, or the speaker's unique vocal quirks. To achieve this, the network acts as an information filter, systematically stripping away low-level acoustic details as the signal propagates through its layers, leaving only abstract linguistic meaning at the final output.

For audio forensics and deepfake detection, this filtering process creates a fundamental paradox. The subtle technical glitches left behind by generative algorithms—such as vocoder irregularities, phase mismatches, and artificial silence boundaries—are precisely the types of "noise" that semantic models are trained to discard. As a result, the final layers of state-of-the-art speech models often become a forensic graveyard.

To catch a deepfake, researchers have discovered that the key lies not in the final semantic outputs, but in the intermediate and shallow representations of self-supervised learning (SSL) models.

The Layer-Wise Hierarchy of Speech Models

Like computer vision models that transition from detecting simple edges to complex objects, self-supervised speech models (such as wav2vec 2.0, HuBERT, and WavLM) process audio through a strict structural hierarchy.

[Raw Audio] ──> [Shallow Layers] ───────> [Intermediate Layers] ───> [Deep Layers]
                 • Spectrogram defects     • Jitter & shimmer        • Phonemes
                 • Mute segments           • Vocoder artifacts       • Semantic meaning
                 • Phase anomalies         • Speaker identity        • Linguistic content

In the initial transformer layers, the network encodes raw physical properties of the acoustic signal. As Yi et al. demonstrate, these shallow network layers are highly effective at capturing fine-grained acoustic artifacts, such as spectrogram defects and unnatural mute segments.

As the signal moves into the intermediate layers, the model begins to organize these raw signals into localized acoustic patterns. Lee et al. show that these mid-level representations serve as the "sweet spot" for detecting synthetic speech because they preserve micro-anomalies like jitter (short-term variations in frequency) and shimmer (short-term variations in amplitude). These subtle traces are characteristic of neural vocoders but are completely absent from organic human speech.

By the time the representation reaches the deepest layers, the focus shifts entirely to high-level semantic abstractions. Han et al. emphasize that features optimized for speech-to-text tasks prioritize linguistic content over the signal generation process itself. Consequently, deep layers actively filter out the very acoustic nuances required to identify generative forgeries.

This divergence between acoustic detail and semantic abstraction is not unique to deepfake detection. Even in benign speech applications, forcing a model to use its final layers can be counterproductive for non-linguistic tasks. For example, Zheng et al. found that the information required for speaker diarisation ("who spoke when") steadily diminishes in the final layers of models optimized for speech recognition, leading to higher error rates if forced to use the final semantic embeddings.

Why Deeper and Bigger Models Fail Forensics

In many deep learning applications, scaling up a model's size or extracting features from its final layer yields a strict performance benefit. In audio forensics, however, deeper representations can introduce counterproductive noise.

When investigating how automatic speech recognition (ASR) models scale, Salvi et al. observed that while larger models initially improve deepfake detection, performance eventually plateaus or declines at the largest scales. The highly specialized linguistic representations in massive models become so invariant to signal-level properties that they lose their forensic utility.

Forensic Utility
  ▲
  │       ▲ [Intermediate Layers]
  │      ╱ ╲  (Captures jitter, shimmer, vocoder artifacts)
  │     ╱   ╲
  │    ╱     ▼ [Deep/Semantic Layers]
  │   ╱         (Introduces semantic noise, discards signal details)
  └───┴────────────────────────────────────────────────────────► Layer Depth

This behavior occurs because deep layers treat algorithmic artifacts as irrelevant variations. El Kheir et al. discovered that the critical heavy lifting for deepfake detection occurs early in the network—specifically within transformer layers 4 through 12. Pushing deeper into the network introduces semantic noise that obscures technical glitches.

Similarly, when dealing with partially forged audio where only specific segments are altered, Ji et al. note that deep semantic layers obscure the sharp boundary inconsistencies and signal irregularities that occur at the transition points between real and synthetic speech.

Strategies for Artifact Preservation

To exploit these insights, recent architectures bypass the semantic bottleneck of deep layers using three main strategies: truncation, fusion, and distillation.

Layer Truncation and Selection

The most direct approach is to simply discard the deeper half of a pre-trained model. By restricting feature extraction to early and intermediate layers, detectors can achieve higher accuracy while slashing computational overhead. El Kheir et al. demonstrated that cutting off deep layers yields superior deepfake detection rates while significantly reducing inference time.

Attentive Merging and Fusion

Rather than selecting a single layer, models can dynamically combine representations from across the entire network hierarchy to form a multi-view perspective.

  • Attentive Merging: Liu et al. introduced an Attentive Merging (AttM) mechanism that learns to weight and combine the internal hidden layers of a WavLM model. Their work confirmed that utilizing only the first half of the model's layers provides a more effective and efficient representation for anti-spoofing than using the entire network.
  • Layer Fusion: For environmental sound forensics, Guo et al. implemented a lightweight layer-fused system that aggregates intermediate representations from a frozen audio encoder, capturing the low-level signatures of generative processes before they are stripped away by the final layers.
  • Retrieval Augmentation: Taking multi-layer feature extraction a step further, Kang et al. developed a retrieval-augmented framework. By querying an external database of genuine audio using features from multiple depths, their system uses shallower layers to catch timbral and acoustic inconsistencies, while leveraging deeper layers to verify abstract semantic patterns.

Knowledge Distillation

When computational constraints prevent the deployment of large, multi-layer fusion networks, knowledge distillation can compress artifact-awareness into compact models.

Yi et al. utilized a self-distillation method where a deep teacher network guides shallower student networks, effectively transferring fine-grained, shallow-level acoustic details down the architecture. Alternatively, Lee et al. proposed Representation Selective Self-distillation (RSSD), which forces the network to explicitly focus on the intermediate layer "sweet spot," maximizing the model's sensitivity to vocoder-induced irregularities.

The Complementary Role of Semantics

While low-level acoustic artifacts are highly effective for identifying current generation deepfakes, a robust forensic framework cannot completely ignore high-level structure. As generative models continue to improve, their ability to perfectly mimic low-level signal characteristics will inevitably advance.

However, as Salvi et al. point out, deepfake generators that excel at mimicking low-level signal properties often still struggle to replicate high-level semantic features, such as long-term speaker identity consistency or natural facial-vocal coordination in multimodal settings. The future of audio forensics relies on carefully balancing these representations—ensuring that in the pursuit of semantic meaning, we do not discard the physical reality of the signal.

Key Foundational Work

Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification (2022) — contributed 3 concept elements to this direction.
Tandem Multitask Training of Speaker Diarisation and Speech Recognition for Meeting Transcription (2022) — contributed 1 concept elements to this direction.
Learning From Yourself: A Self-Distillation Method For Fake Speech Detection (2023) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_acoustic_artifact_preservation_1f0424ba,
  title = {acoustic artifact preservation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/acoustic-artifact-preservation/rdr_1f0424baf6ef4a28bf9c8858ce2aa0d3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_acoustic_artifact_preservation_1f0424ba_v1,
  title = {acoustic artifact preservation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/acoustic-artifact-preservation/rdr_1f0424baf6ef4a28bf9c8858ce2aa0d3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake