Semantic Sequence Compression via Connectionist Temporal Classification

29 concepts22 papers2 proposals

Modern speech processing faces a fundamental structural conflict: the "length mismatch" between audio and text. While a single second of audio can generate hundreds of high-frequency feature frames, the corresponding text contains only a handful of tokens. As Papi et al. note in Speechformer: Reducing Information Loss in Direct Speech Translation, this redundancy makes the $O(n^2)$ memory complexity of Transformer architectures a significant bottleneck for direct speech-to-text systems.

To bridge this gap, researchers have converged on a mechanism that uses Connectionist Temporal Classification (CTC) not just for final prediction, but as a dynamic filter to collapse redundant audio frames into a condensed semantic representation.

The Intuition: CTC as a Semantic Sieve

The core idea relies on the "peaky" nature of CTC. When a model is trained with a CTC loss, it learns to predict specific tokens (phonemes, characters, or subwords) at precise temporal locations, filling the remaining frames with a "blank" token.

This behavior provides a natural roadmap for sequence compression. As described by Dong et al. in Consecutive Decoding for Speech-to-text Translation, the system can identify phoneme boundaries by removing these blank frames and merging consecutive identical predictions. By collapsing a run of identical frames into a single vector, the model decouples semantic content from temporal duration. Harwath et al. demonstrate in Text-Free Image-to-Speech Synthesis Using Learned Segmental Units that this allows the model to focus on content selection while leaving timing to a separate synthesis module.

Bridging the Modality Gap

The most powerful application of this technique is in aligning speech with Large Language Models (LLMs). Because LLMs are trained on discrete text tokens, they struggle to process the raw, over-sampled output of a speech encoder.

Recent work has moved toward "word-boundary-aware" or "subword-level" compression to achieve a near 1:1 ratio between speech units and text tokens. For instance:

Efficiency and Performance Gains

Beyond semantic alignment, this compression serves as a critical efficiency hack. By pruning redundant frames mid-inference, models can significantly reduce the computational load on the decoder.

The authors of FFM: A Frame Filtering Mechanism To Accelerate Inference Speed For Conformer In Speech Recognition found that content-aware pruning can reduce sequence length by up to 73%, leading to a 34% reduction in real-time factor with minimal impact on accuracy. Similarly, Tsunoo et al. demonstrate in Decoder-only Architecture for Streaming End-to-end Speech Recognition that CTC-compressed audio prompts allow decoder-only architectures to run twice as fast as standard encoder-decoder baselines.

Beyond Hard Pruning: Soft Alignment

While CTC provides a discrete path for compression, some researchers augment this with "soft" similarity-based methods. Xu et al. propose in Optimal Transport Regularization for Speech Text Alignment in Spoken Language Models using Optimal Transport to merge adjacent frames with high semantic similarity. This ensures that even in the absence of explicit token boundaries, the model prunes non-informative segments like silence while retaining essential linguistic features.

Whether through hard CTC filtering or soft similarity merging, the goal remains the same: transforming a high-frequency acoustic signal into a sparse, semantically dense sequence that a language model can understand as easily as text.

Key Foundational Work

Consecutive Decoding for Speech-to-text Translation (2021) — contributed 2 concept elements to this direction.
CTC-based Compression for Direct Speech Translation (2021) — contributed 1 concept elements to this direction.
Speechformer: Reducing Information Loss in Direct Speech Translation (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_ctc_based_sequence_compression_8d17edf7,
  title = {CTC-based sequence compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/ctc-based-sequence-compression/rdr_8d17edf7da9f4dbbbad45676626abde7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_ctc_based_sequence_compression_8d17edf7_v1,
  title = {CTC-based sequence compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/ctc-based-sequence-compression/rdr_8d17edf7da9f4dbbbad45676626abde7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake