Deep autoregressive models represent the state-of-the-art for density modeling in domains such as audio (WaveNet) and image generation. However, their sequential nature ($O(T)$ steps) makes them computationally expensive for real-time applications. To address this, researchers often employ probability distillation, transferring knowledge from a slow "teacher" model to a fast, parallelizable "student" model. While standard practice relies on minimizing the reverse Kullback-Leibler (KL) divergence, this approach frequently fails in high-dimensional spaces, leading to model collapse and poor sample quality. This paper investigates the theoretical roots of this failure and proposes robust reconstruction-based alternatives.
The Curse of Dimensionality in Distillation
The primary challenge in distilling high-dimensional models using reverse-KL divergence is what the authors term the "curse of dimensionality." In theory, the reverse-KL objective should encourage the student to cover the teacher's probability mass. This requires an "expansion signal"—a gradient that pushes the student's distribution to widen when it is too narrow.
However, the authors prove that as the dimensionality of the data increases, the probability of receiving a helpful expansion signal becomes a "rare event." In high-dimensional spaces, the student model often gets stuck in a tiny region of the teacher's distribution because the mathematical gradients for exploration vanish. This explains why student models in speech synthesis often produce "whispering" artifacts; they have collapsed into a low-entropy state that fails to capture the full variance of the teacher.
From Evaluative to Instructive Learning
The authors categorize standard reverse-KL distillation as evaluative: the student generates a sample, and the teacher provides a score. If the teacher's distribution is highly structured, the student's random guesses are almost always "wrong," providing no useful signal for improvement. To overcome this, the paper proposes two instructive alternatives that treat the teacher as an encoder:
- $z$-reconstruction: This method minimizes distance in the latent space. A student's sample is passed through the teacher's inverse function to see if it can reconstruct the original noise source.
- $x$-reconstruction: The teacher acts as an encoder and the student as a decoder. By taking real data samples, encoding them with the teacher, and training the student to reconstruct that specific data, the teacher "shows" the student exactly how to map latent points to the data manifold.
These methods bypass the need for the student to "stumble" upon the correct distribution, anchoring the training process to the empirical data distribution ($p_D$).
Experimental Results
The proposed reconstruction methods were validated across toy models, image datasets (MNIST, CIFAR-10), and neural vocoder tasks. In toy experiments with multivariate Gaussians, the reconstruction methods successfully matched the target distributions where reverse-KL failed.
In practical applications, $x$-reconstruction produced sharp, recognizable images on MNIST and Fashion-MNIST, whereas KL-based models produced blurry or striped artifacts. In speech synthesis, the reconstruction approach eliminated the common "whispering" artifacts without requiring the manual, task-specific heuristics (like power loss) typically needed for Parallel WaveNet. Notably, on CIFAR-10, the student model trained with $x$-reconstruction even demonstrated greater global coherence than its own teacher model.
Conclusion
While reverse-KL divergence is the standard objective for probability distillation, its "evaluative" nature makes it ill-suited for high-dimensional generative modeling. By shifting to "instructive" reconstruction-based objectives, researchers can achieve more stable training and higher-quality synthesis. Future work may involve combining these reconstruction losses with perceptual priors to further refine the fidelity of distilled models.
Summary updated