TL;DR
This thesis tackles the critical challenge of uncertainty quantification in deep sequence models by integrating them with probabilistic methods. It introduces novel techniques for Transformer architectures and online learning, and explores self-supervised learning for generative models, ultimately bridging the gap between these two powerful AI paradigms.
What Problem Does This Paper Address?
Deep sequence models, like Transformers, excel at prediction but often lack awareness of their own uncertainty. This is a major hurdle for deployment in safety-critical applications. Traditional probabilistic models, particularly Bayesian methods, offer a principled way to quantify uncertainty but struggle with scalability when applied to complex deep learning architectures. The paper addresses two key bottlenecks: specifying appropriate priors and achieving high-quality approximate inference in deep neural networks. It also investigates how to improve generative models by leveraging self-supervised signals, inspired by the success of diffusion models.
Methods
The paper proposes several novel methods:
- Sparse Gaussian Process Attention (SGPA): This technique calibrates Transformer uncertainty by connecting the attention mechanism to sparse Gaussian processes (GPs). It replaces the standard softmax operation in attention with a kernel-based approach, allowing for uncertainty estimation. Decoupled SGPA is introduced to improve computational efficiency.
- Online HiPPO Sparse Variational Gaussian Process (OHSVGP): This method leverages the HiPPO framework's long-range memory capabilities to improve online Bayesian inference in sparse GP models. It interprets HiPPO's projections as interdomain inducing variables, enabling efficient online updates and mitigating catastrophic forgetting in continual learning.
- Pseudo Video Generation for Generative Models: Inspired by diffusion models, this approach uses data augmentation to create "pseudo videos" from images. These videos provide self-supervised signals for intermediate latent states, improving the performance of generative models like VQVAE and DDPM when extended to video generation. The paper also theoretically analyzes the benefits of higher-order Markov chains for creating these pseudo videos.
Datasets
The paper evaluates its methods on a variety of standard datasets:
- Image Classification: CIFAR10, CIFAR100, and corrupted versions (CIFAR10-C, CIFAR100-C).
- Natural Language Processing: CoLA (linguistic acceptability) and IMDB (sentiment analysis).
- Time Series Prediction: Solar Irradiance, Audio Signal, COVID-19 death counts, and Powerplant data.
- Graph Property Regression: ZINC dataset.
- Video Generation: CIFAR10 and CelebA datasets are used to create pseudo videos.
No new benchmarks are introduced; the focus is on applying existing datasets to novel model architectures and training paradigms.
Key Insights and Ideas
- Attention as a GP Posterior Mean: A key insight is the equivalence between kernel-based attention and the posterior mean of a sparse Gaussian process. This connection naturally leads to using GP priors for attention outputs and SGPA for uncertainty quantification in Transformers.
- HiPPO for Online Learning Memory: The HiPPO framework's ability to capture long-range dependencies is crucial for online learning. By interpreting HiPPO's projections as interdomain inducing variables, OHSVGP can effectively maintain memory of past data, preventing catastrophic forgetting.
- Self-Supervision from Pseudo Videos: Diffusion models' success can be partly attributed to self-supervised signals for intermediate latent states. The paper demonstrates that creating pseudo videos from images and using them to train generative models can inject similar beneficial inductive biases, improving generation quality.
- Higher-Order Markov Chains for Pseudo Videos: First-order Markov chains for pseudo video generation can be suboptimal. The paper theoretically shows that higher-order Markov chains can provide more expressive and informative pseudo videos, leading to better generation.
Key Results
- SGPA: Transformers with SGPA achieve competitive predictive accuracy while significantly improving in-distribution uncertainty calibration and out-of-distribution robustness and detection. For example, SGPAE (ensemble) outperforms DE (ensemble) in calibration.
- OHSVGP: Outperforms existing online sparse GP methods in scenarios requiring long-term memory, demonstrating robust performance in continual learning tasks. It also offers significant computational efficiency gains over OSVGP.
- Pseudo Video Generation: Improves image generation quality for VQVAE and DDPM counterparts, with 4-frame and 8-frame pseudo videos showing better results than 1-frame models. The performance gain is empirically justified by the self-supervised signals.
Limitations and Future Work
- SGPA: The paper notes that standard SGPA can be computationally inefficient for Transformers. While decoupled SGPA addresses this, further computational cost reduction for large language models is desirable. Extending SGPA to decoder-based Transformers for autoregressive tasks is also a key future direction.
- OHSVGP: The performance of OHSVGP in continual learning depends on the sorting criterion used for data. Finding optimal sorting strategies for real-world problems is an open question.
- Pseudo Video Generation: The design space for creating informative pseudo videos is vast, and finding optimal data augmentation strategies for different models and architectures remains an open research question. Applying this framework to other data modalities (text, molecules) is also an interesting future direction.
Summary updated