Progressive Label Distillation: Learning Input-Efficient Deep Neural Networks

Zhong Qiu Lin, Alexander WongarXiv 2019 · OpenReview

Abstract

Much of the focus in the area of knowledge distillation has been on distilling knowledge from a larger teacher network to a smaller student network. However, there has been little research on how the concept of distillation can be leveraged to distill the knowledge encapsulated in the training data itself into a reduced form. In this study, we explore the concept of progressive label distillation, where we leverage a series of teacher-student network pairs to progressively generate distilled training data for learning deep neural networks with greatly reduced input dimensions. To investigate the efficacy of the proposed progressive label distillation approach, we experimented with learning a deep limited vocabulary speech recognition network based on generated 500ms input utterances distilled progressively from 1000ms source training data, and demonstrated a significant increase in test accuracy of almost 78% compared to direct learning.

Introduction: The Challenge of Input Efficiency

Deep Learning models are notoriously power-hungry, making them difficult to deploy on "edge" devices like smartphones or IoT sensors. While most research focuses on model distillation—shrinking the number of parameters in a network—this paper targets input efficiency.

The computational cost of a network is heavily tied to the size of the data fed into it. However, simply cropping data (e.g., feeding a 500ms snippet of a 1-second "yes" command) creates a "noisy label" problem. If you crop the first half of a word, the audio might just be silence, but the ground-truth label still says "yes." Training a model on these mismatched pairs leads to poor performance. The authors sought a way to reduce input dimensions while maintaining the semantic integrity of the training labels.

Methodology: Progressive Label Distillation

The authors propose a novel framework to bridge the gap between high-resolution source data and low-resolution target inputs.

1. Label Distillation

Instead of using the original ground-truth labels for a cropped piece of data, they use a "Teacher" network (trained on full-sized data) to predict what is actually in the crop. If a crop of a "yes" clip is actually silent, the Teacher will label it as "silence," providing a much more accurate signal for the "Student" network to learn from.

2. Progressive Steps

Rather than jumping directly from a large input (1000ms) to a small one (500ms), they use a chain of networks. A 1000ms teacher trains an 800ms student; that 800ms student then becomes the teacher for a 600ms student, and so on. This gradual reduction helps the network adapt to the loss of information more gracefully.

The model architecture takes MFCC features derived from audio signals and processes them through a series of residual blocks.

The student networks are trained using Soft Labels (probability distributions) or Hard Labels (the single most likely class). Soft labels are particularly effective when the gap between the teacher's input size and the student's input size is large, as they provide "dark knowledge"—nuance about shared characteristics between classes that helps the student learn even when the input signal is sparse.

Experimental Setup

The study utilized the Speech Command dataset, a common benchmark for limited-vocabulary speech recognition, containing 105,829 one-second utterances across 35 different words. Audio was converted into 40-dimensional Mel-frequency cepstrum coefficient (MFCC) representations before being fed into a res15 (deep residual) architecture.

Key Results and Insights

The researchers compared several training strategies for a 500ms input target:

  • Direct Learning (No Distillation): 12.03% accuracy. The model essentially fails to learn because the ground-truth labels are too noisy for the cropped data.
  • Direct Label Distillation (1000ms → 500ms): 85.82% accuracy. A massive improvement by simply using a teacher to re-label the crops.
  • Progressive Label Distillation (1000ms → 800ms → 500ms): 89.22% accuracy. This two-step progression yielded the best results.
Test accuracy across different input dimensions (C500-C800) as a function of the number of progressive distillation steps.

As shown in the figure above, there is a "Goldilocks Zone" for progression. Moving in steps is better than one giant leap, but using too many intermediate steps (e.g., 5 steps to get to 500ms) causes accuracy to drop, likely due to "generational loss" or error compounding.

Efficiency Gains

The 500ms model (C500) required only 776M FLOPs, compared to the 1.64B FLOPs required by the 1000ms model—a 53% reduction in computational cost with only a ~6% drop in absolute accuracy compared to the full-sized teacher.

Conclusion

Progressive Label Distillation offers a powerful way to create input-efficient models for edge deployment. By treating the teacher network as an automated, high-fidelity annotator for sub-sections of data, researchers can significantly reduce the computational footprint of deep neural networks. Future work will explore applying this technique to other domains like computer vision and optimizing the selection of intermediate distillation steps.

Summary updated

Abstract

Much of the focus in the area of knowledge distillation has been on distilling knowledge from a larger teacher network to a smaller student network. However, there has been little research on how the concept of distillation can be leveraged to distill the knowledge encapsulated in the training data itself into a reduced form. In this study, we explore the concept of progressive label distillation, where we leverage a series of teacher-student network pairs to progressively generate distilled training data for learning deep neural networks with greatly reduced input dimensions. To investigate the efficacy of the proposed progressive label distillation approach, we experimented with learning a deep limited vocabulary speech recognition network based on generated 500ms input utterances distilled progressively from 1000ms source training data, and demonstrated a significant increase in test accuracy of almost 78% compared to direct learning.
BibTeX
@misc{lin2019progressive,
  author = {Zhong Qiu Lin and Alexander Wong},
  title = {{Progressive Label Distillation: Learning Input-Efficient Deep Neural Networks}},
  year = {2019},
  eprint = {1901.09135},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1901.09135},
}
Figure