Interpret Your Decision: Logical Reasoning Regularization for Generalization in Visual Classification

Zhaorui Tan, Xi Yang, Qiufeng Wang, Anh Nguyen, Kaizhu Huang

Introduction

Deep learning models are notoriously brittle, often failing when encountering data from unseen domains or entirely new object categories. This failure of generalization typically stems from models latching onto spurious correlations, such as background textures or lighting, rather than the core semantic features of an object. To address this, researchers have introduced L-Reg (Logical Regularization), a novel regularization term that integrates formal logical reasoning into the deep learning framework. L-Reg forces models to focus on "minimal yet sufficient" semantic features—such as a face for a person—improving both classification accuracy and interpretability across complex real-world data shifts.

The paper identifies two primary hurdles in generalization: Multi-Domain Generalization (mDG), which involves classifying known objects in entirely new environments, and Generalized Category Discovery (GCD), which requires identifying known classes while simultaneously clustering and identifying "unknown" classes. The most challenging setting, termed "All-shift," combines both problems.

Comparison of Multi-domain generalization, Generalized category discovery, and the combined All-shift scenario.

Methodology: Logical Regularization

L-Reg is a plug-and-play loss function derived from a logical analysis framework. Unlike standard regularization techniques like $L_2$ weight decay, which simply shrink weights, L-Reg reshapes the latent feature space to be more balanced and "dividable."

Key technical components include:

  • Semantic Support: L-Reg encourages the model to identify the "semantic support"—the minimal combination of features necessary to deduce a class. For example, a "guitar" is defined by its fingerboard; if that's present, the logic holds.
  • Entropy-Based Regularization: Formulated using conditional entropy, it minimizes the complexity of the feature distribution and classifier weights, preventing over-reliance on extreme feature values.
  • Atomic Formulas: The relationship between an image and its label is treated as a logical "atomic formula" ($h(g(x)) \to y$). The regularization ensures these formulas remain valid even under significant data shifts.

Experimental Setup

The effectiveness of L-Reg was evaluated across a wide range of benchmarks:

  • mDG Benchmarks: PACS (Photos, Art, Cartoons, Sketches), VLCS, OfficeHome, TerraIncognita, and DomainNet.
  • GCD Benchmarks: Fine-grained datasets like CUB (birds), Stanford Cars, and Herbarium19, along with generic ones like CIFAR10/100 and ImageNet-100.
  • Specialized Tasks: The method was even tested on CircuitNet for congestion prediction in circuit design, proving the logic-based approach works beyond standard image pixels.

Key Results and Interpretability

L-Reg consistently outperformed established baselines across all tasks. In the challenging "All-Shift" scenario, L-Reg improved performance on unknown classes by 7.01% over the GMDG baseline. On the TerraIncognita dataset, average accuracy jumped from 60.7% to 62.9%. In GCD tasks, unknown class accuracy on the Herbarium19 dataset increased from 34.8% to 37.2%.

A significant advantage of L-Reg is the inherent interpretability it provides. By forcing the model to use minimal semantics, it naturally focuses on the "right" features. GradCAM visualizations demonstrate that while standard models might focus on ambiguous regions or backgrounds, L-Reg-equipped models focus on definitive features like human faces to identify people.

GradCAM visualizations comparing models without and with L-Reg. Models with L-Reg focus more precisely on core semantic features like faces.

Discussion and Limitations

The research reveals that complexity is often the enemy of generalization. L-Reg reduces model complexity by balancing feature distributions and reducing the number of extreme classifier weights. However, a notable limitation occurs in abstract domains like Sketches. Because L-Reg seeks specific semantic supports (e.g., detailed facial features), it can struggle when those features are represented by simple geometric shapes or stick figures.

Future work will focus on improving latent representations to better handle abstract domains and mitigating the slight performance trade-offs sometimes observed in known classes when prioritizing unknown class discovery.

Paper Citation: Tan, Z., Yang, X., Wang, Q., Nguyen, A., & Huang, K. (2024). Interpret Your Decision: Logical Reasoning Regularization for Generalization in Visual Classification. 38th Conference on Neural Information Processing Systems (NeurIPS 2024).

Summary updated

Abstract

Vision models excel in image classification but struggle to generalize to unseen data, such as classifying images from unseen domains or discovering novel categories. In this paper, we explore the relationship between logical reasoning and deep learning generalization in visual classification. A logical regularization termed L-Reg is derived which bridges a logical analysis framework to image classification. Our work reveals that L-Reg reduces the complexity of the model in terms of the feature distribution and classifier weights. Specifically, we unveil the interpretability brought by L-Reg, as it enables the model to extract the salient features, such as faces to persons, for classification. Theoretical analysis and experiments demonstrate that L-Reg enhances generalization across various scenarios, including multi-domain generalization and generalized category discovery. In complex real-world scenarios where images span unknown classes and unseen domains, L-Reg consistently improves generalization, highlighting its practical efficacy.
BibTeX
@inproceedings{tan2024interpret,
  author = {Zhaorui Tan and Xi Yang and Qiufeng Wang and Anh Nguyen and Kaizhu Huang},
  title = {{Interpret Your Decision: Logical Reasoning Regularization for Generalization in Visual Classification}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=Woiqqi5bYV},
}
Figure