Vector Quantized Bayesian Neural Network Inference for Data Streams

Namuk Park, Taekyu Lee, Songkuk KimAAAI 2021 · DOI

Introduction

Standard deterministic neural networks, while highly accurate, are notoriously overconfident and lack the ability to quantify predictive uncertainty. Bayesian Neural Networks (BNNs) address this by modeling a distribution over network weights, which is essential for safety-critical domains like autonomous driving. However, BNNs typically rely on Monte Carlo (MC) sampling—running dozens of forward passes for a single input—making them roughly 50 times slower than deterministic counterparts. This computational overhead creates a significant bottleneck for real-time data streams, such as high-frequency video processing.

The VQ-BNN Approach

To bridge the gap between BNN reliability and deterministic speed, we introduce Vector Quantized BNN (VQ-BNN) inference. The core innovation is a "predict once and remember" strategy that exploits the temporal redundancy inherent in data streams. Instead of performing 50 forward passes on a single frame, VQ-BNN performs only one forward pass per frame using a single weight sample. It then compensates for the missing samples by leveraging a buffer of "memorized" predictions from previous frames.

Comparison of standard BNN inference (multiple passes per input) versus VQ-BNN inference (one pass per input, leveraging memorized past predictions).

Key Components

  • Prototype Memory: A small buffer stores recent inputs ($x$) and their corresponding predictions ($p(y|x, w)$), each generated with a different random weight sample.
  • Temporal Exponential Smoothing: The final predictive distribution is calculated as a weighted sum of the current prediction and these memorized prototypes.
  • Importance Weighting: The contribution of past predictions decays exponentially over time ($\pi \propto \exp(-|t|/\tau)$). This assumes that in a continuous stream, a prediction from a few milliseconds ago serves as a valid proxy for a different weight sample on the current frame.

Experimental Evaluation

VQ-BNN was evaluated across three distinct domains: synthetic linear regression for visualization, the CamVid dataset for semantic segmentation, and NYUDv2 for monocular depth estimation.

Performance and Speed

In semantic segmentation tasks, VQ-BNN achieved a 33x speedup over standard BNNs in batch processing and a 12x speedup in stream processing. Remarkably, it was only 7–10% slower than a non-Bayesian deterministic network. On the CamVid dataset, VQ-BNN not only matched but slightly exceeded BNN accuracy (92.0% vs. 91.8%) and achieved a lower Negative Log-Likelihood (0.253 vs. 0.276), indicating better-calibrated uncertainty.

Key Insights and Discussion

VQ-BNN's ability to approximate the full BNN distribution using temporal proxies allows it to maintain high-fidelity uncertainty estimates without the iterative cost. As shown in the visualization below, VQ-BNN (c) captures a distribution of predictions similar to a full BNN (b), whereas a standard DNN (a) provides only a single point estimate.

Visualization of predictive distributions for DNN, BNN, and VQ-BNN. VQ-BNN successfully approximates the multi-modal uncertainty of a full BNN.

Robustness and Limitations

Beyond speed, VQ-BNN demonstrates enhanced robustness to noise. By averaging predictions over time, it smooths out transient errors that might cause a single-frame BNN to produce erratic results. However, the method is inherently dependent on temporal consistency. In scenarios with frequent, abrupt scene cuts or jumps in the data stream, the relevance of memorized prototypes diminishes, potentially degrading performance. Additionally, while computationally efficient, the model requires a small amount of extra memory to maintain the prototype buffer.

Conclusion

VQ-BNN offers a practical path forward for deploying Bayesian uncertainty in real-time applications. By reframing the inference problem to exploit temporal context, it achieves the speed of deterministic networks with the robust uncertainty quantification of Bayesian methods.

Summary updated

BibTeX
@inproceedings{park2021vector,
  author = {Namuk Park and Taekyu Lee and Songkuk Kim},
  title = {{Vector Quantized Bayesian Neural Network Inference for Data Streams}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i10.17124},
}
Figure