Contrastive Reasoning in Neural Networks

Mohit Prabhushankar, Ghassan AlRegib

TL;DR

Standard deep networks classify objects by looking for matching patterns (induction), making them fragile to noise and out-of-distribution distortions. This paper introduces contrastive reasoning, an abductive approach that evaluates multiple hypotheses by asking "Why class $P$ rather than class $Q$?". By extracting these implicit contrastive features via backpropagated gradients, the authors significantly boost model robustness against unseen distortions without retraining the base network.

What Problem Does This Paper Address?

Deep neural networks excel at standard classification but remain notoriously fragile when test data is corrupted or shifted (e.g., by blur, noise, or environmental changes). This vulnerability stems from their reliance on inductive reasoning—mapping features directly from cause to effect based purely on correlations found in the training data. If the input domain changes slightly, these feed-forward patterns break down.

To bridge this generalization gap, the authors look to human visual perception, which frequently employs abductive reasoning (inferring cause from effect). When a human looks at a noisy image of a spoonbill, they don't just look for spoonbill features; they also actively recognize that it lacks the S-shaped neck of a flamingo and the white body of a crane. The authors formalize this "counter-hypothesis" testing as contrastive inference to make deep networks inherently more robust.

Fig. 1. Discriminative features are identified in Feed-Forward reasoning frameworks. Contrasts between observed facts and a known knowledge base is used as reasons in Contrastive Inference.

Methods

Instead of relying solely on standard feed-forward activations, the proposed framework extracts a network's implicit contrastive knowledge using gradients:

  1. Contrastive Features via Gradients: For a network's initial prediction $P$ and any alternative class $i$, the authors compute the gradient of a loss function with respect to the weights: $$r_i = \nabla_{W_l} J(W, x, P, i)$$ This gradient captures the exact structural model change required to alter the network's prediction from $P$ to $i$.

  2. Modified Loss Function: Rather than standard cross-entropy, they introduce a modified Mean Squared Error (MSE) loss function that uses a Kronecker delta scaled by the mean training logits. This forces the network to evaluate its contrastive confidence accurately.

  3. The Inference Pipeline: The individual contrast gradients for all $N$ possible classes are concatenated into a single rich feature vector $r_x$. A lightweight Multi-Layer Perceptron (MLP), denoted as $\mathcal{H}$, is trained on these pristine contrastive features. The final prediction combines the traditional feed-forward output and $\mathcal{H}$'s output through a non-Bayesian abductive formulation.
  4. Contrastive Explanations: The authors also adapt Grad-CAM to use these contrastive gradients, visually highlighting the exact regions that differentiate class $P$ from class $Q$.

Datasets

The authors evaluated their approach across several standard and corrupted image recognition benchmarks:

  • CIFAR-10: Used as the pristine baseline training set.
  • CIFAR-10-C: A benchmark containing 19 different global and local distortions (e.g., blur, noise, pixelation) across 5 severity levels.
  • STL-10: Used to test higher-resolution images ($96 \times 96$) under simulated noise and blur corruptions.
  • VisDA: A challenging synthetic-to-real domain adaptation dataset consisting of 12 object classes across 152k images.

Key Insights and Ideas

  • Implicit Knowledge is Already There: Pre-trained networks already possess deep contrastive knowledge embedded within their weights; we do not need extra data to generate it, only gradients to unlock it.
  • Effect-to-Cause Reasoning: Instead of asking "What features make this a spoonbill?", contrastive reasoning asks "What prevents this from being something else?". Ruling out alternative classes provides a safety net when the primary class features are obscured by noise.
  • Machine Logic vs. Human Logic: Contrastive visualizations reveal that networks often rule out classes based on geometric features humans might ignore. For instance, a network distinguishes a "No-Left Turn" sign from a "Stop" sign by checking for the absence of a hexagonal corner in the bottom left of the image.

Key Results

  • Robustness to Distortions: On CIFAR-10-C, the contrastive framework achieved an average accuracy improvement of 3.47% across all 19 distortions, with performance jumping by 8.22% specifically on glass blur corruptions.
  • Domain Shifts & Resolution: The method yielded a 2.56% average accuracy boost on noisy STL-10 images and a 5.48% overall improvement on the synthetic-to-real VisDA dataset.
  • Zero Overhead on Clean Data: When tested on pristine data, the contrastive framework naturally reduces to standard feed-forward performance, ensuring no degradation in baseline accuracy.

Limitations and Future Work

  • Sensitivity to Statistical Distortions: The performance gains are lower (<3%) on distortions that alter global image statistics, such as brightness and contrast changes. Because early layers filter out these variations, gradients from the final layer alone are insufficient to capture them. The authors suggest that deriving contrastive features from earlier layers is a vital next step.
  • Dependence on Initial Predictions: The abductive formulation relies on the initial feed-forward prediction $P$. If the base network is overly confident in an incorrect prediction, it can adversely affect the subsequent contrastive inference step.

Summary updated

Abstract

Neural networks represent data as projections on trained weights in a high dimensional manifold. The trained weights act as a knowledge base consisting of causal class dependencies. Inference built on features that identify these dependencies is termed as feed-forward inference. Such inference mechanisms are justified based on classical cause-to-effect inductive reasoning models. Inductive reasoning based feed-forward inference is widely used due to its mathematical simplicity and operational ease. Nevertheless, feed-forward models do not generalize well to untrained situations. To alleviate this generalization challenge, we propose using an effect-to-cause inference model that reasons abductively. Here, the features represent the change from existing weight dependencies given a certain effect. We term this change as contrast and the ensuing reasoning mechanism as contrastive reasoning. In this paper, we formalize the structure of contrastive reasoning and propose a methodology to extract a neural network's notion of contrast. We demonstrate the value of contrastive reasoning in two stages of a neural network's reasoning pipeline : in inferring and visually explaining decisions for the application of object recognition. We illustrate the value of contrastively recognizing images under distortions by reporting an improvement of 3.47%, 2.56%, and 5.48% in average accuracy under the proposed contrastive framework on CIFAR-10C, noisy STL-10, and VisDA datasets respectively.
BibTeX
@misc{prabhushankar2021contrastive,
  author = {Mohit Prabhushankar and Ghassan AlRegib},
  title = {{Contrastive Reasoning in Neural Networks}},
  year = {2021},
  eprint = {2103.12329},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.12329},
}
Figure