TL;DR
Star Temporal Classification (STC) is a novel loss function designed to allow sequence models to learn from partially labeled data—where an unknown number of tokens are missing at unknown positions. By introducing a special "star" wildcard token within a weighted finite-state transducer (WFST) framework, STC enables training for speech and handwriting recognizers that nearly matches fully supervised performance, even when up to 70% of the labels are missing.
What Problem Does This Paper Address?
In traditional temporal classification tasks like Automatic Speech Recognition (ASR), models typically rely on loss functions like Connectionist Temporal Classification (CTC). CTC is highly effective because it does not require explicit segmentation (knowing exactly when each word was said). However, it strictly requires the full sequence of words in the correct order.
Real-world data is rarely this clean. Datasets often contain "weak labels" where only high-confidence words are transcribed, or historical documents where portions of text are illegible. As shown below, a partial label might skip several words entirely.
Standard CTC breaks down in these scenarios because it cannot account for these gaps. The authors address this limitation by creating a loss function that explicitly models the possibility of missing tokens anywhere in the sequence.
Methods
The core innovation of this paper is the Star Temporal Classification (STC) algorithm. It builds upon the WFST-based implementation of CTC but introduces a special star token ($\langle*\rangle$).
The Star Token and Graph
In the STC label graph, a star token is inserted between every known label. This star acts as a wildcard that can represent zero or more tokens from the entire vocabulary alphabet. To prevent the model from getting confused and counting the same alignment multiple times, the authors utilize a "relative complement" token ($\langle*\rangle \setminus t$), which represents any token except the one that just occurred.
Training Stability: Token Insertion Penalty ($\lambda$)
Simply allowing wildcards everywhere makes the early stages of training highly unstable; the model might attempt to "explain" the entire input using only star tokens to minimize effort. To resolve this, the authors introduce a token insertion penalty ($\lambda$). This adds a cost to using the star token, which is gradually decayed during training (using an exponential decay scheme) as the model becomes more confident in predicting actual labels.
Implementation
The system is implemented using the GTN framework, which allows for automatic differentiation through WFST operations. To make this computationally feasible for large vocabularies (like the 50k words in LibriSpeech), the authors optimized the data transfer between the GPU (where the neural network runs) and the CPU (where the WFST composition happens). They achieved this by only moving gradients for tokens present in the partial label and the star tokens.
Datasets
The authors evaluated STC on two primary benchmarks:
- LibriSpeech (ASR): 960 hours of English speech. They simulated partial labels by randomly dropping words with a probability $p\_drop$ ranging from 0.1 to 0.7.
- IAM Handwriting Database (HWR): A benchmark for offline handwriting recognition. Here, they dropped characters instead of words to test the method's versatility across different temporal classification tasks.
Key Insights and Ideas
- Wildcards as Regularizers: By allowing the model to "skip" parts of the signal it isn't sure about (via the star token), STC effectively focuses training on the high-certainty labels provided.
- Bridging the Gap to Full Supervision: One of the most significant findings is that STC doesn't just outperform a broken CTC; it actually recovers most of the performance of a fully supervised model.
- Efficiency Matters: Without the specific optimization of transferring only relevant token probabilities between GPU and CPU, training on a 50,000-word vocabulary would be roughly 18x slower.
- Pseudo-labeling Synergy: STC serves as an excellent "first pass" teacher. A model trained with STC on partial labels can be used to generate full "pseudo-labels" for the dataset. A standard CTC model can then be trained on those pseudo-labels to achieve even better results.
Key Results
- Speech Recognition: On LibriSpeech, even when 70% of the words were missing ($p\_drop = 0.7$), STC achieved a Word Error Rate (WER) within ~1% of the fully supervised baseline (when combined with pseudo-labeling). In contrast, standard CTC fails completely at high drop rates.
- Handwriting Recognition: STC demonstrated similar resilience in HWR. At a 50% character drop rate, STC maintained a 13.5% Character Error Rate (CER), while the CTC baseline ballooned to 53.6%.
- Runtime: Despite the added complexity of the star tokens, the optimized STC implementation is only about 20% slower per epoch than standard CTC on the LibriSpeech task.
Limitations and Future Work
- Hyperparameter Sensitivity: The token insertion penalty ($\lambda$) and its decay rate ($\tau$) are critical for success and require careful tuning based on the expected amount of missing data ($p\_drop$).
- Noisy Labels: While STC handles missing labels exceptionally well, it doesn't explicitly handle incorrect labels (noise). The authors suggest that extending the WFST framework to handle substitutions or insertions in the transcript is a promising future direction.
- Specialization: The authors note that the label graph could be further specialized. For example, the star token could be restricted to represent a specific range of missing characters (e.g., "between 2 and 4 characters missing") if that prior knowledge is available.
Summary updated