Contextual Reliability: When Different Features Matter in Different Contexts

Gaurav R. Ghosal, Amrith Setlur, Daniel S. Brown, Anca D. Dragan, Aditi Raghunathan

Abstract

Standard approaches to robust machine learning often categorize features into a binary: "reliable" features that generalize across domains and "spurious" features that lead to shortcut learning. This paper challenges this dichotomy, arguing that feature reliability is inherently context-dependent. We introduce Explicit Non-spurious feature Prediction (ENP), a two-stage framework that learns to dynamically identify reliable features based on the current context. By utilizing minimal human supervision to train a feature predictor, ENP achieves state-of-the-art performance across image classification, reinforcement learning, and real-world autonomous driving tasks.

Introduction

Deep neural networks are highly susceptible to "shortcut learning," where models rely on spurious correlations—such as identifying a cow based on a grassy background—that fail when the environment changes. To combat this, existing robust machine learning methods typically enforce invariance, instructing the model to ignore spurious features entirely.

However, this paper identifies a fundamental flaw in the invariance paradigm: the "correct" features to utilize often vary depending on the specific context. Consider an autonomous vehicle. Generally, the speed of a car in an adjacent lane is a spurious feature for the ego-vehicle's braking decisions. However, if the ego-vehicle is approaching a crosswalk and the neighboring car suddenly brakes, that car's speed becomes a critical, reliable cue indicating a hidden pedestrian. Forcing a model to be invariant to the neighboring lane's speed in all scenarios would make the system less safe. This research formalizes this challenge as Contextual Reliability.

The ENP Framework

To address context-dependent reliability, the authors propose Explicit Non-spurious feature Prediction (ENP). This framework moves away from end-to-end training in favor of a structured, two-stage approach:

Stage 1: Feature Prediction

Instead of predicting the final target immediately, the system first trains a "feature predictor." This component analyzes the input and generates a mask that identifies which features are reliable for the current latent context. This stage is trained using a small subset of data (typically 0.5% to 10%) annotated by humans to indicate feature relevance. Because identifying what to look at is often a simpler task than solving the entire prediction problem, the predictor can achieve high accuracy (90%+) even with limited supervision.

Stage 2: Target Model Training

The primary model is then trained to make the final prediction (e.g., determining a vehicle's trajectory). During this stage, the model is forced to be invariant to any features labeled as "spurious" by the Stage 1 predictor. This is achieved through data augmentation, where noise is added to or features are zeroed out if they are deemed unreliable in that specific context. This ensures the model learns to rely only on the contextually appropriate signals.

Experimental Evaluation

The researchers evaluated ENP across three distinct domains to test its versatility:

  1. Corrupted Waterbirds: In this modified image benchmark, the foreground (the bird) is occasionally blurred, making the background (traditionally a spurious feature) necessary for identification. ENP achieved 72.8% worst-case accuracy, matching the performance of models that had access to significantly more context information.
  2. Noisy MountainCar: In this reinforcement learning simulation, previous actions are usually misleading but become vital cues when the primary state sensor is noisy. ENP achieved a reward of -139.5, significantly outperforming the best baseline of -170.4.
  3. Waymo Open Motion Dataset (WOMD): In a real-world autonomous driving application, ENP used human-provided causal labels to identify which nearby agents influenced a driver's behavior. ENP improved trajectory prediction accuracy (minADE) by 5% over standard models, demonstrating scalability to high-dimensional, real-world data.

Key Insights

The Spurious/Reliable Binary is a Myth

The core contribution of this work is the realization that robustness is not about finding a single set of "true" features. Rather, it is about dynamic feature selection. A feature that is a "shortcut" in one scenario may be a "signal" in another.

Annotation Efficiency

ENP breaks the "chicken-and-egg" problem of context—where a model needs context to find features but needs features to understand context—by using minimal human intervention. The efficiency of the feature predictor suggests that human experts do not need to label entire datasets; they only need to provide the "rules of engagement" for a small fraction of the data.

Preventing Overfitting

Unlike methods such as Context-based Distributionally Robust Optimization (conDRO), which often overfit to minority contexts, ENP maintains stability by sharing the definition of "reliable" features across contexts while explicitly masking out noise.

Limitations and Future Work

While ENP shows significant promise, it relies on human ability to identify causal or reliable features. In highly abstract domains where human intuition fails, the method's efficacy may be limited. Additionally, the current framework assumes features can be separated (e.g., foreground vs. background). Applying ENP to the highly entangled latent spaces of large-scale pre-trained models remains an area for future research. The authors also suggest exploring "test-time interventions," where human experts could correct the feature predictor in real-time for high-stakes decision-making.

Summary updated

BibTeX
@inproceedings{ghosal2023contextual,
  author = {Gaurav R. Ghosal and Amrith Setlur and Daniel S. Brown and Anca D. Dragan and Aditi Raghunathan},
  title = {{Contextual Reliability: When Different Features Matter in Different Contexts}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=s1hrcLUcld},
}
Figure