A Novel Technique for Evidence based Conditional Inference in Deep Neural Networks via Latent Feature Perturbation

Dinesh Khandelwal, Suyash Agrawal, Parag Singla, Chetan AroraarXiv 2018 · arXiv

This paper, titled "A Novel Technique for Evidence based Conditional Inference in Deep Neural Networks via Latent Feature Perturbation," presents a clever way to make deep learning models "smarter" at test time by incorporating external evidence—like image tags or captions—without needing specialized multi-modal training or massive jointly-labeled datasets.

TL;DR

The authors propose a Bayesian framework that treats inference as a test-time optimization problem. By perturbing a model's latent feature representation to maximize the probability of observed evidence (e.g., "this image contains a dog"), they significantly boost the accuracy of tasks like semantic and instance segmentation using standard pre-trained backbones.

What Problem Does This Paper Address?

In many real-world scenarios, we have "auxiliary information" available at inference time. For example, if you are segmenting an image, you might already have user-provided tags or a caption. Ideally, a model should use this "evidence" to refine its predictions.

However, standard Deep Neural Networks (DNNs) are typically rigid feed-forward systems. Incorporating extra info usually requires:

  1. Specialized Architectures: Designing multi-modal "fusion" networks.
  2. Jointly Labeled Data: Massive datasets where every image has both the primary label (e.g., a pixel-mask) and the auxiliary label (e.g., a caption).
  3. Handling Sparse Evidence: DNNs often ignore low-dimensional evidence (like a single tag) in favor of high-dimensional image features, a problem known as "mode collapse" in some contexts.

This paper aims to bypass these hurdles, allowing any standard model to ingest arbitrary evidence at runtime to improve its primary task.

Methods

The core idea is to frame the problem as Multi-Task Learning (MTL) combined with Test-Time Optimization.

1. The MTL Architecture

The model consists of a shared "trunk" (encoder) that produces a latent representation $z$. This $z$ is then fed into two branches:

  • Primary Task: The main goal (e.g., Semantic Segmentation).
  • Auxiliary Task: A task that predicts the evidence (e.g., Image Classification for tags).
The MTL architecture showing shared layers and task-specific branches.

2. Bayesian Latent Perturbation

Instead of a simple forward pass, the authors use a Bayesian formulation to find the best output $y$ given input $x$ and evidence $a$. They show that this is equivalent to finding a latent representation $z$ that maximizes the probability of the evidence.

At test time, they perform a few steps of backpropagation—not to update the whole model, but to perturb the latent features. They propose two flavors:

  • Z-Direct (ZD): Directly updating the latent vector $z$.
  • Z-Weight (ZW): Updating the weights of the shared trunk ($W_{zx}$) to better align the latent space with the evidence.

To prevent the model from "hallucinating" to satisfy the evidence, they use regularization techniques like L2-norm penalties or early stopping.

The process of backpropagating loss from the auxiliary task at test time to update the trunk weights.

Datasets

The authors evaluated their framework on several standard benchmarks:

  • PASCAL VOC 2012: Used for semantic segmentation, augmented with additional annotations.
  • MS COCO: Used for both image captioning and instance segmentation.
  • Unconstrained Web Data: The authors also tested on random images from the web to demonstrate the framework's robustness to real-world, "in-the-wild" evidence.

Key Insights and Ideas

  • Evidence as Prediction: By modeling evidence as the output of a secondary task, you can use standard pre-trained models (like a ResNet classifier) as the auxiliary branch. This eliminates the need for complex multi-modal training from scratch.
  • Inference as Optimization: The "aha moment" is treating the latent space as a flexible medium. If the evidence says there is a "dog" but the model is unsure, the framework "nudges" the latent features until the model's internal state strongly reflects the presence of a dog.
  • Object Discovery: A striking qualitative finding is that this method can "discover" objects that the baseline model completely missed, simply because the evidence provided a hint that the object should be there.
Figure 1: The figure shows some sample results by our frame- work exploiting the commonly available evidence at the test time. The 1st row shows the output of instance segmentation exploiting corresponding textual description, whereas the 2nd row gives se- mantic segmentation output using image level tags as the evidence (auxiliary information). Both the images as well as auxiliary in- formation are deliberately sourced from unconstrained data on the web (and not the benchmark datasets) to show easy availability of such information at the test time.

Key Results

The framework consistently outperformed state-of-the-art (SOTA) baselines across multiple tasks:

  • Semantic Segmentation: Achieved a 3.9% mIoU improvement over the SOTA DeepLabv3+ on PASCAL VOC when using image tags as evidence.
  • Instance Segmentation: Showed a 2.8% improvement (relative) when using image captions as evidence.
  • Image Captioning: Improved the BLEU-4 score from 25.8 to 29.6 by incorporating image tags.
  • Robustness: The method remained effective even when "noisy" (incorrect) tags were added to the evidence, degrading much more gracefully than naive pruning strategies.

Limitations and Future Work

  • Computational Overhead: Because the method requires a few iterations of backpropagation at test time, inference is slower. For example, DeepLabv3+ takes 0.15s per image, while this method takes ~2.6s.
  • Overfitting to Evidence: If the optimization runs for too many iterations, the model can overfit to the evidence and start predicting objects that aren't actually in the image (e.g., predicting multiple surfboards just because the caption mentions "waves").
  • Ground Truth Inconsistencies: The authors noted that their quantitative gains were sometimes capped because the model correctly identified objects that the human annotators had missed in the "ground truth" datasets. Future work could involve using this technique to help clean or verify large-scale datasets.

Summary updated

Abstract

Auxiliary information can be exploited in machine learning models using the paradigm of evidence based conditional inference. Multi-modal techniques in Deep Neural Networks (DNNs) can be seen as perturbing the latent feature representation for incorporating evidence from the auxiliary modality. However, they require training a specialized network which can map sparse evidence to a high dimensional latent space vector. Designing such a network, as well as collecting jointly labeled data for training is a non-trivial task. In this paper, we present a novel multi-task learning (MTL) based framework to perform evidence based conditional inference in DNNs which can overcome both these shortcomings. Our framework incorporates evidence as the output of secondary task(s), while modeling the original problem as the primary task of interest. During inference, we employ a novel Bayesian formulation to change the joint latent feature representation so as to maximize the probability of the observed evidence. Since our approach models evidence as prediction from a DNN, this can often be achieved using standard pre-trained backbones for popular tasks, eliminating the need for training altogether. Even when training is required, our MTL architecture ensures the same can be done without any need for jointly labeled data. Exploiting evidence using our framework, we show an improvement of 3.9% over the state-of-the-art, for predicting semantic segmentation given the image tags, and 2.8% for predicting instance segmentation given image captions.
BibTeX
@misc{khandelwal2018novel,
  author = {Dinesh Khandelwal and Suyash Agrawal and Parag Singla and Chetan Arora},
  title = {{A Novel Technique for Evidence based Conditional Inference in Deep Neural Networks via Latent Feature Perturbation}},
  year = {2018},
  eprint = {1811.09796},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.09796},
}
Figure