TL;DR
Deep sequence recognition models often become overconfident, leading to unreliable predictions. This paper introduces a novel framework, PSSR, that leverages perceptually and semantically similar sequences to regularize training, significantly improving confidence calibration across various sequence recognition tasks.
What Problem Does This Paper Address?
Deep sequence recognition (DSR) models, widely used in applications like scene text recognition (STR) and speech recognition (SR), have achieved impressive accuracy. However, a critical issue is their tendency towards overconfidence. This means the models assign high confidence scores to their predictions, even when those predictions are incorrect. This overconfidence can be detrimental in safety-critical applications where reliable uncertainty estimates are crucial. Existing methods often rely on simple label smoothing, which treats all tokens equally and doesn't fully exploit the rich correlations present in sequential data. The paper argues that this lack of consideration for related sequences leads to suboptimal calibration.
Methods
The core of this work is the Perception and Semantic aware Sequence Regularization (PSSR) framework. PSSR introduces a new regularization term to the standard loss function that incorporates information from "similar" sequences.
Here's a breakdown of the key components:
- Similar Sequence Mining:
- Perception Similarity Sequences: To capture visual or auditory similarities, the authors use a semantic context-free model (like a CRNN) to generate candidate sequences. The sequences with the highest probability from this model are considered perceptually similar. This helps identify cases where visually similar characters (e.g., 'l' and 'i') might lead to overconfidence.
- Semantic Correlation Sequences: To capture contextual relationships, a pre-trained language model (like BCN) is used. This model helps identify sequences that are semantically coherent with the target sequence, even if they differ visually. This addresses cases where semantically related but incorrect words might be predicted with high confidence.
- Sequence-level Calibration: Unlike token-level calibration methods, PSSR operates on entire sequences, avoiding the need for complex token-to-token alignments.
- Adaptive Calibration Intensity: Recognizing that overconfidence varies with sample difficulty, PSSR introduces a modulating factor function. This function adaptively adjusts the calibration intensity based on the posterior probability of the target sequence (inspired by Focal Loss), giving more attention to harder samples.
The overall loss function is defined as: $L_{total} = L_{G}(Y_i, \hat{Y}_i) + \alpha f(p_i) \sum_{Y \in S(X_i, Y_i)} L_{G}(Y, Y_i)$ where $L_G$ is the empirical loss, $\alpha$ is a global calibration intensity hyperparameter, $f(p_i)$ is the adaptive calibration intensity function, and $S(X_i, Y_i)$ is the set of similar sequences.
Datasets
The paper evaluates PSSR on two primary sequence recognition tasks:
- Scene Text Recognition (STR):
- English Benchmark: Trained on synthetic datasets (Synth90K, SynthText) and tested on a combination of seven realistic datasets (IIIT5K, SVT, IC03, IC13, IC15, SVTP, CUTE80).
- Chinese Benchmark: Trained and tested on a large-scale dataset comprising five public datasets.
- Speech Recognition (SR):
- AISHELL-1: A large-scale Mandarin speech dataset.
The paper also mentions experiments on corrupted datasets to test robustness to dataset shift.
Key Insights and Ideas
- Dual Nature of Overconfidence: The paper clearly distinguishes between perception overconfidence (due to visual similarity) and semantic overconfidence (due to contextual similarity), providing concrete examples like 'l' vs. 'i' and "universiti" vs. "university".
- Value of Similar Sequences: The core insight is that sequences with high perceptual or semantic correlation to the target sequence are not just noise but valuable sources of information for regularization. They can help the model learn to be less certain about visually or semantically ambiguous predictions.
- Sample Difficulty Matters: Overconfidence is not uniform across all samples. Harder samples (those with lower target probability) are more prone to overconfidence. The adaptive calibration intensity module directly addresses this by tailoring regularization based on sample hardness.
- Sequence-level vs. Token-level: Operating at the sequence level simplifies the process and makes the method more broadly applicable to different DSR architectures without requiring explicit token alignments.
Key Results
PSSR demonstrates significant improvements in confidence calibration across various benchmarks and models:
- STR: PSSR consistently outperforms state-of-the-art methods (including GLS and CASLS) on both attention-based and CTC-based models for English and Chinese STR. For example, on the English STR benchmark with TRBA (attention-based), PSSR achieved ECE, ACE, and MCE reductions of 0.56%, 0.62%, and 3.18% compared to GLS. For CTC-based models, PSSR significantly reduced ECE, ACE, and MCE.
- SR: On the AISHELL-1 dataset, PSSR reduced ECE by 20.54% and ACE by 20.69% for attention-based models, and by 17.73% and 17.85% for CTC-based models, respectively.
- Robustness: PSSR shows strong calibration performance even under dataset shift conditions with various corruption types.
- Active Learning: The calibrated confidence from PSSR improved active learning query strategies, leading to higher accuracy compared to using uncalibrated confidence.
Limitations and Future Work
- Computational Cost: While PSSR aims to improve calibration, mining similar sequences and incorporating them into the loss function can increase computational overhead. The paper doesn't explicitly detail the exact computational cost increase.
- Hyperparameter Tuning: The effectiveness of PSSR relies on hyperparameters like $\alpha$, $\epsilon_e$, and $\epsilon_h$. Finding optimal values might require careful tuning.
- Future Work: The authors suggest exploring more effective strategies to jointly utilize perception and semantic information for even better DSR model calibration. They also mention verifying the method's effectiveness on downstream applications like active learning, which they have begun to do.
Summary updated