Variational Connectionist Temporal Classification

Linlin Chao, Jingdong Chen, Wei ChuECCV 2020 · OpenReview

Abstract

Connectionist Temporal Classification (CTC) is a training criterion designed for sequence labelling problems where the alignment between the inputs and the target labels is unknown. One of the key steps is to add a blank symbol to the target vocabulary. However, CTC tends to output spiky distributions since it prefers to output blank symbol most of the time. These spiky distributions show inferior alignments and the non-blank symbols are not learned sufficiently. To remedy this, we propose variational CTC (Var-CTC) to enhance the learning of non-blank symbols. The proposed Var-CTC converts the output distribution of vanilla CTC with hierarchy distribution. It first learns the approximated posterior distribution of blank to determine whether to output a specific non-blank symbol or not. Then it learns the alignment between non-blank symbols and input sequence. Experiments on scene text recognition and offline handwritten text recognition show Var-CTC achieves better alignments. Besides, with the enhanced learning of non-blank symbols, the confidence scores of model outputs are more discriminative. Compared with the vanilla CTC, the proposed Var-CTC can improve the recall performance by a large margin when the models maintain the same level of precision.

Introduction

Connectionist Temporal Classification (CTC) is the industry standard for sequence labeling tasks where the alignment between input (such as an image of a word) and output (the text) is unknown. To handle this, CTC introduces a "blank" symbol to fill the gaps between characters. However, CTC has a notorious flaw: spiky distributions. Because the blank symbol is so flexible and appears so frequently in training paths, models become "lazy," preferring to output blanks almost everywhere and producing very sharp, narrow spikes for actual characters. This leads to poor alignment and unreliable confidence scores, making it difficult to filter errors in real-world applications.

Methodology: Variational CTC

To address these limitations, we propose Variational CTC (Var-CTC), which replaces the flat distribution of vanilla CTC with a hierarchical distribution. Instead of picking from the entire vocabulary (including the blank) at once, the model decouples the decision into two steps: determining if a character exists at a timestep (the "Blank" decision, $O_b$) and identifying which character it is (the "Classification" decision, $O_c$).

Architecture of the proposed Variational CTC (Var-CTC) model.

As illustrated in the architecture above, Var-CTC utilizes Variational Inference during training. An approximate posterior ($q$) is introduced that is "label-aware," using ground truth text to guide the model toward better alignments. The training objective maximizes the Evidence Lower Bound (ELBO), which balances the likelihood of predictions with a KL-divergence penalty. This hierarchy forces the model to focus more on the features of actual characters, relieving the pressure caused by the unbalanced distribution of blanks and leading to more stable gradients.

Experimental Results

Var-CTC was evaluated on Scene Text Recognition (STR) benchmarks (including IIIT5K and SVT) and the IAM Handwriting dataset.

Qualitative Alignment

Visualization of the output distributions demonstrates that Var-CTC's character probabilities align much more closely with the actual visual boundaries of characters in the image. In contrast, vanilla CTC produces spiky and often shifted peaks.

Comparison of output distributions between Var-CTC (top) and vanilla CTC (bottom).

Precision and Recall Performance

While Var-CTC achieves comparable raw sequence accuracy to vanilla CTC, its primary strength lies in its Precision-Recall (PR) performance. Because the model learns non-blank symbols more sufficiently, its confidence scores are more discriminative. When maintaining a high precision level of 98%, Var-CTC improved recall by 31.7% to 50.8% across various scene text datasets compared to standard CTC. This makes it significantly more reliable for industrial OCR systems that require high-confidence filtering.

Conclusion

Var-CTC is an efficient, "drop-in" replacement for standard CTC that adds only about 0.3% more parameters and FLOPS. By decoupling existence from identity through a hierarchical variational framework, it solves the spiky distribution problem, resulting in superior alignment and highly reliable confidence scores for sequence labeling tasks.

Summary updated

Abstract

Connectionist Temporal Classification (CTC) is a training criterion designed for sequence labelling problems where the alignment between the inputs and the target labels is unknown. One of the key steps is to add a blank symbol to the target vocabulary. However, CTC tends to output spiky distributions since it prefers to output blank symbol most of the time. These spiky distributions show inferior alignments and the non-blank symbols are not learned sufficiently. To remedy this, we propose variational CTC (Var-CTC) to enhance the learning of non-blank symbols. The proposed Var-CTC converts the output distribution of vanilla CTC with hierarchy distribution. It first learns the approximated posterior distribution of blank to determine whether to output a specific non-blank symbol or not. Then it learns the alignment between non-blank symbols and input sequence. Experiments on scene text recognition and offline handwritten text recognition show Var-CTC achieves better alignments. Besides, with the enhanced learning of non-blank symbols, the confidence scores of model outputs are more discriminative. Compared with the vanilla CTC, the proposed Var-CTC can improve the recall performance by a large margin when the models maintain the same level of precision.
BibTeX
@inproceedings{chao2020variational,
  author = {Linlin Chao and Jingdong Chen and Wei Chu},
  title = {{Variational Connectionist Temporal Classification}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=sA_s7JQAYe},
}
Figure