BAR: Bayesian Activity Recognition using variational inference

Ranganath Krishnan, Mahesh Subedar, Omesh TickooarXiv 2018 · arXiv

TL;DR

Researchers at Intel Labs have introduced BAR (Bayesian Activity Recognition), a framework that brings principled uncertainty estimation to video analysis. By replacing the final layers of a standard 3D ConvNet with variational layers, the model doesn't just guess what’s happening in a video—it quantifies how sure it is, leading to a 6.2% boost in Precision-Recall AUC and significantly better detection of out-of-distribution data.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) have become the gold standard for visual activity recognition, but they have a dangerous "blind spot": they are notoriously overconfident. A standard DNN will output a high Softmax probability for a class even when presented with noisy data or something it has never seen before (out-of-distribution samples).

In safety-critical applications like autonomous surveillance or human-robot interaction, an overconfident wrong answer can be catastrophic. The authors address this by moving away from deterministic "point-estimate" models toward Bayesian Deep Learning, which allows a system to "know what it doesn't know" by providing reliable uncertainty estimates alongside its predictions.

Methods

The authors built their framework on the ResNet-101 C3D architecture, a standard for processing spatiotemporal features in video. The core innovation lies in the final stages of the network:

  1. Variational Layers: They replaced the final three fully connected layers with variational layers. Instead of learning a single fixed weight for each connection, the model learns a probability distribution (a mean-field normal distribution) over the weights.
  2. Stochastic Variational Inference (SVI): To train this, they used SVI to minimize the Kullback-Leibler (KL) divergence, effectively finding a tractable approximation of the true posterior distribution of the model parameters.
  3. Flipout: To keep training efficient, they employed the "Flipout" technique, which decorrelates gradients within a mini-batch by sampling pseudo-independent weight perturbations for each input.
  4. Monte Carlo Sampling: During inference, the model performs 40 stochastic forward passes. By sampling from the learned weight distributions 40 times, the model generates a predictive distribution rather than a single number.
  5. Uncertainty Metrics: They used BALD (Bayesian Active Learning by Disagreement) and Predictive Entropy to quantify model uncertainty. BALD is particularly useful as it captures the mutual information between the parameter posterior and the predictive distribution.

Datasets

The models were evaluated using two major benchmarks:

  • Moments-in-Time (MiT): A large-scale dataset of one-million short videos. The authors selected a subset of 54 classes for "in-distribution" training and a separate subset of 54 classes to act as "out-of-distribution" (OOD) samples.
  • UCF101: A classic action recognition dataset. In some experiments, UCF101 was used as the in-distribution set while MiT samples were treated as OOD to test how the model handles entirely different data sources.

Key Insights and Ideas

  • Softmax is Not Confidence: A major takeaway is that standard Softmax outputs represent relative probability between classes, not absolute model confidence. The paper demonstrates that standard DNNs remain highly confident even when they are wrong.
  • SVI vs. MC Dropout: While Monte Carlo (MC) Dropout is a popular "cheap" way to get uncertainty, this paper shows that Stochastic Variational Inference (SVI) provides more reliable confidence measures and better separation between correct and incorrect predictions.
  • Reliable OOD Detection: The Bayesian approach creates a clear "gap" in the uncertainty histograms. When the model sees a video class it wasn't trained on, the BALD and Entropy scores spike, providing a clear signal that the model’s prediction should not be trusted.

Key Results

  • Performance Boost: The BAR model achieved a 6.2% improvement in Precision-Recall AUC compared to a non-Bayesian baseline and a 3.63% improvement over the MC Dropout method.
  • ROC AUC: The Bayesian SVI model showed a 1.3% improvement in ROC AUC over the standard DNN.
  • Accuracy Preservation: Importantly, adding Bayesian uncertainty did not hurt raw performance; the Top-1 accuracy (54.10%) remained slightly higher than the baseline DNN (53.90%).
  • Qualitative Reliability: Histograms of confidence measures showed that for false predictions, the Bayesian model’s confidence was significantly lower (skewed left) compared to the baseline DNN, which remained stubbornly overconfident (skewed right).

Limitations and Future Work

  • Computational Overhead: The paper requires 40 Monte Carlo samples for every inference. While this provides robust uncertainty, it increases the computational cost by 40x compared to a single deterministic pass, which might be a hurdle for real-time edge applications.
  • Multimodal Expansion: The authors suggest that the next step is extending this framework to multimodal inputs (combining audio and vision). They envision using the uncertainty estimates from each modality to intelligently fuse information—for example, trusting the audio more if the video feed is blurry or noisy.
  • Trimmed vs. Untrimmed Video: This work focused on trimmed video samples; applying this to long-form, untrimmed "in-the-wild" video streams remains a future challenge.

Summary updated

Abstract

Uncertainty estimation in deep neural networks is essential for designing reliable and robust AI systems. Applications such as video surveillance for identifying suspicious activities are designed with deep neural networks (DNNs), but DNNs do not provide uncertainty estimates. Capturing reliable uncertainty estimates in safety and security critical applications will help to establish trust in the AI system. Our contribution is to apply Bayesian deep learning framework to visual activity recognition application and quantify model uncertainty along with principled confidence. We utilize the stochastic variational inference technique while training the Bayesian DNNs to infer the approximate posterior distribution around model parameters and perform Monte Carlo sampling on the posterior of model parameters to obtain the predictive distribution. We show that the Bayesian inference applied to DNNs provide reliable confidence measures for visual activity recognition task as compared to conventional DNNs. We also show that our method improves the visual activity recognition precision-recall AUC by 6.2% compared to non-Bayesian baseline. We evaluate our models on Moments-In-Time (MiT) activity recognition dataset by selecting a subset of in- and out-of-distribution video samples.
BibTeX
@misc{krishnan2018bar,
  author = {Ranganath Krishnan and Mahesh Subedar and Omesh Tickoo},
  title = {{BAR: Bayesian Activity Recognition using variational inference}},
  year = {2018},
  eprint = {1811.03305},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.03305},
}

Loading related content...

Figure