Bidirectional Audio Infilling for Speech Editing

50 concepts30 papers2 proposals

When we listen to natural speech, we are incredibly sensitive to context. A speaker’s pitch, rhythm, and volume do not exist in a vacuum; they form continuous contours that span across words and sentences. Furthermore, every recording carries a unique environmental footprint—a specific blend of background noise, room reverberation, and microphone characteristics.

Traditional Text-to-Speech (TTS) models are designed for sequential, left-to-right generation. While this works well for creating entirely new audio, it fails fundamentally when applied to speech editing—the task of altering, inserting, or deleting specific words within an existing recording. If you use a standard TTS model to generate a replacement word and splice it into a sentence, the result sounds jarring. The inserted segment lacks awareness of the surrounding acoustic environment and the speaker's immediate prosodic trajectory, leading to noticeable pitch jumps, phase mismatches, and boundary glitches.

To solve this, a powerful paradigm has emerged: bidirectional audio infilling. Instead of generating speech from scratch, these models treat speech editing as an audio "inpainting" task. By masking the target region and conditioning the generation process on both the preceding (past) and succeeding (future) acoustic context, these systems synthesize audio that blends seamlessly into the original recording.


Why Left-to-Right Generation Fails at Boundaries

To understand why bidirectional context is necessary, consider what happens at the boundaries of an edit. In a standard causal autoregressive model, the acoustic frame at time $t$ is predicted using only the frames from $1$ to $t-1$.

If we attempt to perform an insertion or replacement using this approach, the model can match the left boundary (the past context) reasonably well. However, it has no mechanism to look ahead at the right boundary (the future context). It does not know if the speaker is about to drop their pitch, pause for breath, or if a sudden background noise occurs right after the edit block. As a result, when the generated segment ends and the original recording resumes, the transition is almost always discontinuous. As noted by the authors of EditSinger, simple concatenation in complex acoustic or musical environments inevitably introduces audible glitches and unnatural pitch jumps.

Bidirectional infilling eliminates this issue by transforming the future context from an unknown variable into a hard constraint. The model must find an acoustic path that satisfies both the history leading up to the edit and the future state immediately following it.


The Core Mechanism: Inpainting the Spectrogram

The most intuitive way to implement this is to treat the audio spectrogram as an image with a missing region. By masking specific frames of a mel-spectrogram or latent representation, generative models can learn to reconstruct the missing content by interpolating the surrounding patterns.

Original Audio:  [--- Preceding Context ---] [--- Target Region ---] [--- Succeeding Context ---]
Masked Input:    [--- Preceding Context ---] [##### MASKED #####] [--- Succeeding Context ---]
Target Text:                                 "New inserted text"

Tang et al. demonstrated this in Zero-Shot Text-to-Speech for Text-Based Insertion in Audio Narration, using a Transformer decoder to capture long-range dependencies across both sides of the mask. Because the model processes the entire sequence globally, the inserted words naturally inherit the rhythm, energy, and flow of the original recording.

This bidirectional awareness yields several powerful emergent properties:


Architectural Strategies for Bidirectional Infilling

How do modern architectures natively incorporate both past and future context? Researchers have converged on two primary methodologies: Token Rearrangement and Iterative Diffusion/Flow-Matching.

1. Token Rearrangement (Prefix-Suffix-Middle)

Autoregressive language models are inherently causal (left-to-right). To force a causal model to look into the future, researchers use a clever sequence manipulation trick: they cut out the target region and paste it at the very end of the sequence.

In VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild, Peng et al. introduced a token rearrangement procedure that restructures the input stream. Similarly, Zheng utilized a "prefix-suffix-middle" reordering strategy in VoiceCraft-X: Unifying Multilingual, Voice-Cloning Speech Synthesis and Speech Editing.

Instead of processing the audio chronologically, the sequence is fed into the model as: $$\text{Sequence} = [\text{Prefix}] \parallel [\text{Suffix}] \parallel [\text{Middle}]$$

Because the [Suffix] (the future context) appears before the [Middle] (the target edit region) in the sequence, a standard causal attention mechanism can condition on both the past and the future while generating the replacement tokens. Mohammad et al. successfully extended this sequence-completion framework to linear-time state-space models in Speak, Edit, Repeat: High-Fidelity Voice Editing and Zero-Shot TTS with Cross-Attentive Mamba.

2. Non-Autoregressive Masked Modeling and Diffusion

Non-autoregressive architectures do not generate audio frame-by-frame; instead, they refine the entire sequence simultaneously. This makes them a natural fit for infilling.

Le et al. bypassed causal constraints entirely in Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale by utilizing a flow-matching objective trained on bidirectional infilling. During training, random segments of audio are masked, and the model learns to predict them given the remaining audio and text.

Diffusion-based approaches, such as those proposed by Liu et al. (DiffVoice: Text-to-Speech with Latent Diffusion) and Gao et al. (E3 TTS: Easy End-to-End Diffusion-based Text to Speech), treat infilling as an inverse problem. The unmasked context frames are kept fixed as ground truth throughout the reverse diffusion process, while the masked region is iteratively denoised from random noise. This allows the model to flexibly handle edits even when the replacement text requires a significantly different duration than the original segment.


Smoothing the Join Points

Even with bidirectional context, ensuring a perfectly smooth transition at the exact sample where the original audio meets the synthesized audio requires explicit architectural care. Several techniques are used to "force-smooth" these boundaries:


Applications Beyond Simple Editing

The ability to reconstruct missing audio based on surrounding context has unlocked solutions to several long-standing problems in speech processing:

Speech Correction and Therapy

Instead of re-recording an entire sentence when a speaker mispronounces a word, models can selectively mask only the problematic phonemes. Kreuk et al. leveraged this in Correcting Mispronunciations in Speech using Spectrogram Inpainting, showing that localized inpainting preserves the speaker's unique vocal identity and prosody far better than full voice cloning or voice conversion pipelines.

Automated Audio Restoration

In spontaneous speech, audio is often degraded by stutters, dropouts, or heavy background noise. Jiang et al. developed FluentSpeech: Stutter-Oriented Automatic Speech Editing with Context-Aware Diffusion Models to automatically detect and mask disfluencies, using context-aware duration and pitch predictors to bridge the resulting gaps smoothly. For heavily degraded audio, Zen et al. showed in Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representations that text transcripts can act as a powerful linguistic prior, allowing the model to "inpaint" phonemes that are completely missing or obscured by environmental noise.

Non-Speech and Musical Domains

The underlying intuition of bidirectional infilling extends beyond human speech. Axel Roebel demonstrated in A Two Level Montage Approach to Sound Texture Synthesis with Treatment of Unique Events that environmental sound textures (like rain or fire) can be repaired by searching for segments that optimally bridge preceding and following acoustic units. In the symbolic domain, Cheuk et al. applied diffusion-based inpainting to music transcription in DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capability, filling in missing or corrupted musical notes by analyzing the surrounding harmonic context.

Key Foundational Work

A Two Level Montage Approach to Sound Texture Synthesis with Treatment of Unique Events (2014) — contributed 2 concept elements to this direction.
Improving the Multi-Modal Post-Editing (MMPE) CAT Environment based on Professional Translators' Feedback (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bidirectional_context_infilling_a3253c03,
  title = {bidirectional context infilling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-context-infilling/rdr_a3253c03cedf468980f2da5a9628cf11},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_bidirectional_context_infilling_a3253c03_v1,
  title = {bidirectional context infilling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-context-infilling/rdr_a3253c03cedf468980f2da5a9628cf11/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake