Reconstructing Long-Duration Gaps in Audio Signals

67 concepts41 papers1 proposal

When an audio packet is dropped during a Voice over IP (VoIP) call or a streaming session, the immediate solution is Packet Loss Concealment (PLC). Traditional PLC techniques rely on the assumption of local stationarity—the idea that audio signals, especially speech, do not change drastically over a few milliseconds. If a 10ms or 20ms packet is lost, a system can repeat the previous waveform pitch period, perform short-term linear predictive coding (LPC) extrapolation, or apply phase alignment and time-warping to smoothly blend the synthetic patch with incoming packets, as explored by Zopf et al. in their work on time-warping and re-phasing.

However, when network congestion causes bursty or extended packet drops lasting 100 milliseconds to several seconds, this assumption completely breaks down. Over longer durations, the correlation between past acoustic context and future signals decays. Simple repetition or extrapolation produces metallic, robotic artifacts or unnatural silences, destroying conversational flow and speech intelligibility. Early attempts to handle bursty packet loss or signal masking often relied on interleaving data packets to break up long bursts into smaller, isolated drops, combined with reconstruction techniques like MMSE estimation with Vector Quantization (VQ) replicas (Gómez et al. in their study on distributed speech recognition) or missing feature theory (James et al. in their comparison of compensation methods).

While these early methods helped downstream speech recognizers handle missing features, they could not synthesize high-fidelity, continuous audio for human listeners. When signal dropouts are severe or frequency bands are completely lost due to structural network damage, as detailed by Liu et al. in their work on speech signal improvement, traditional signal-level extrapolation fails. To bridge these long-duration gaps, the problem must shift from signal-level interpolation to semantic-level inpainting and generation.

From Extrapolation to Generative Hallucination

To handle long gaps where the exact original signal is fundamentally unrecoverable, modern approaches treat PLC as a conditional generation task. Instead of trying to mathematically extend the waveform, generative models learn the underlying distribution of human speech or music to "hallucinate" a highly plausible replacement segment that matches the speaker's identity, tone, and environment.

Several deep learning architectures have been introduced to achieve this:

  • Adversarial Autoencoders: Pascual et al. developed the Packet Loss Adversarial Auto-Encoder (PLAAE), which moves away from slow, autoregressive step-by-step predictions. By processing audio in parallel and leveraging adversarial training, the model synthesizes coherent blocks of missing audio that naturally transition into the surrounding context.
  • Time-Frequency GANs: Rather than working solely in the time domain, Aironi et al. adapted image-to-image translation frameworks to operate on audio spectrograms. Their bin2bin framework uses specialized audio loss functions and discriminators to maintain structural and perceptual naturalness across long gaps exceeding 100ms.
  • Diffusion Models: Diffusion frameworks, which iteratively denoise a random signal into structured audio, have also been adapted for signal restoration. For instance, Zhang et al. modified a diffusion-based vocoder by coupling it with a deep convolutional inversion network in their speech restoration model, allowing the model to synthesize missing acoustic data lost during severe signal degradation.

Bridging the Semantic Gap with Contextual Cues

While generative models can synthesize natural-sounding speech, they face a fundamental limitation when gaps exceed 250ms: without external information, it is impossible to guess what specific words or musical structures were supposed to occur. To maintain semantic coherence over extended durations, models must incorporate top-down contextual constraints.

Linguistic and Sequential Modeling

One approach is to leverage the sequential and grammatical structure of language. Early work by Lee et al. utilized statistical n-gram models to predict lost speech parameters based on the sequence of previously received frames. For much longer gaps, Kuroiwa et al. combined speech recognition with HMM-based speech synthesis in their lost speech reconstruction method. By treating the missing audio as a hidden variable within Missing Feature Theory—a concept that traces back to the cognitive phenomenon of phonemic restoration during temporal masking by loud sounds (Nakatani et al. in their multi-agent auditory system)—the system infers the most likely phoneme sequence across the silence. This coupling of identification and reconstruction was further refined by Ma et al. using Speech Fragment Decoding.

To capture deep semantic context without explicit text or video, Xue et al. introduced Contrast-PLC, which leverages contrastive learning to extract robust semantic representations of speech. By learning high-level contextual features rather than just low-level acoustic patterns, the model significantly enhances its ability to bridge extended packet loss durations. More recently, Borsos et al. introduced SpeechPainter, a text-conditioned speech inpainting model that uses explicit text transcripts to guide the synthesis of missing words, enabling the seamless restoration of long, unconditioned gaps.

Multimodal Guidance

When text is unavailable, visual cues from video streams provide a powerful alternative. Since lip and facial movements are tightly coupled with acoustic production, they can dictate the linguistic content of missing speech even during total audio dropouts. Zhou et al. introduced Vision-Infused Deep Audio Inpainting (VIAI), treating spectrograms as images and synchronizing them with visual cues like instrument or mouth movements to restore audio gaps up to 0.8 seconds.

Similarly, Morrone et al. in their deep learning framework and Montesinos et al. in their context-based speech synthesis model leveraged deep audio-visual speech representations (such as AV-HuBERT) to guide transformer-based synthesis networks, successfully reconstructing intelligible speech for gaps as long as 1.6 seconds.

[Acoustic Context] ---> [Generative Model] ---> Plausible Tone & Identity
                                 ^
                                 | (Guided by)
[Visual/Textual Context] --------+------------> Exact Semantic Content

Long-Range Structural Graphs

For non-speech audio like music, where linguistic models do not apply, structural repetition provides the necessary context. Holighaus et al. addressed gaps lasting several seconds in their audio inpainting approach by constructing a similarity graph across the entire track. By identifying recurring musical patterns elsewhere in the song (such as a previous chorus or instrumental loop), the algorithm can copy and seamlessly patch structurally consistent content into the missing segment.

Feature-Domain and Multi-Band Architectures for Real-Time Streaming

While large generative and multimodal models work well for offline audio editing, real-time communication systems operate under strict latency and computational constraints. To deploy advanced concealment networks on edge devices, researchers have developed highly optimized architectures.

Instead of repairing the final waveform or high-resolution spectrogram, models can operate directly within the compressed latent space of neural audio codecs. Xue et al. proposed FD-PLC, a feature-domain concealment module that repairs corrupted latent representations before they reach the decoder. This approach maintains speech naturalness during consecutive packet losses up to 120ms while avoiding the robotic artifacts of traditional methods.

Furthermore, processing full-band audio uniformly can be computationally wasteful. Sun et al. introduced BS-PLCNet, which splits the audio into separate frequency bands to model low-frequency pitch structures and high-frequency details independently, incorporating multi-task learning and multi-discriminators.

To make this approach even more efficient for real-time streaming, Xia et al. developed BS-PLCNet 2, utilizing an intra-model knowledge distillation framework. During training, a student network (which only sees past context) is trained to mimic a teacher network that has access to future frames. This allows the causal, real-time model to benefit from non-causal, look-ahead insights during training, achieving superior reconstruction quality with a fraction of the parameter count during live deployment.

Key Foundational Work

A packet loss concealment method using recursive linear prediction (2004) — contributed 3 concept elements to this direction.
A comparison of packet loss compensation methods and interleaving for speech recognition in burst-like packet loss (2004) — contributed 1 concept elements to this direction.
On a n-gram model approach for packet loss concealment (2004) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_packet_loss_in_real_time_communication_230fc774,
  title = {packet loss in real-time communication},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/packet-loss-in-real-time-communication/rdr_230fc774a28245248b44ce639c68f8ce},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_packet_loss_in_real_time_communication_230fc774_v1,
  title = {packet loss in real-time communication},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/packet-loss-in-real-time-communication/rdr_230fc774a28245248b44ce639c68f8ce/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake