PatchGen: Learning Soft Intra-Image Predictive Subsets for Visual Generalization

Zhaorui Tan, Weimiao Yu, Xi YangarXiv 2026 · arXiv

TL;DR

PatchGen is a novel, text-free plug-in module that significantly improves visual generalization by identifying and isolating a sample-adaptive "predictive subset" of image patches while filtering out misleading background context. By focusing strictly on core predictive evidence, it achieves state-of-the-art performance across data shifts, target shifts, and combined shift scenarios without requiring text supervision or prompt engineering.

What Problem Does This Paper Address?

When visual classifiers encounter out-of-distribution data, they frequently stumble. This failure usually stems from a reliance on spurious correlations—contextual elements or background shortcuts that repeatedly co-occur with a label in the training set but change or vanish in the real world. For instance, a model might learn to identify a tumor based on surrounding inflammation tissue simply because they often appear together in the training data, even though inflammation is a non-specific response that can happen without a tumor.

While existing domain generalization techniques attempt to enforce domain invariance (making features look similar across domains), they often fail to isolate the exact, sufficient evidence within an image. Furthermore, while vision-language models like CLIP offer robust transfer capabilities, they depend heavily on precise text descriptions, making them less effective in category-discovery settings where novel classes lack reliable textual labels. PatchGen addresses this gap by focusing on intra-image predictive sufficiency entirely within the visual domain.

Methods

The authors propose a structural hypothesis: every image can be partitioned into an unobserved oracle intra-image predictive subset (the essential evidence) and a complementary context subset (non-essential or redundant regions).

To find this proxy, PatchGen computes cross-patch interaction scores via a multi-head attention mechanism to generate a sample-dependent soft mask ($m_\phi$). The network splits the image representation into a selected branch and a complementary branch.

To ensure this mask actually captures the true predictive evidence, PatchGen is jointly optimized using three specialized objectives alongside the main task loss:

  1. Low-score mask suppression ($\mathcal{L}_{ms}$): Pushes weak mask weights toward zero, sharpening the contrast between essential and non-essential patches without enforcing a rigid, fixed sparsity.
  2. Selected-confidence regularization ($\mathcal{L}_{conf}$): Prevents the mask from becoming trivially sparse by ensuring the selected subset retains enough information to make a highly confident prediction.
  3. Class-conditional feature alignment ($\mathcal{L}_{sim}$): Aligns representations of the same class across different domains to ensure domain-stable feature learning.
Figure 2: Overview of PatchGen. (a) The structural hypothe- sis. (b) PatchGen learns a soft mask as a task-driven proxy for the oracle intra-image predictive subset.

Datasets

The authors evaluated PatchGen across three distinct generalization tasks using several prominent benchmarks:

  • Natural Images (Multi-Domain Generalization): Evaluated on the DomainBed suite, including PACS, VLCS, OfficeHome, TerraIncognita, and DomainNet.
  • Histopathological Images: Evaluated on HISTOPANTUM (a binary normal/tumor dataset across four organs) and HISTOCOLON—a challenging new partial-label benchmark introduced by the authors that combines three distinct colon histopathology datasets (CRC-TP, K-16, K-19) to simulate real-world domain and label shifts.
  • Category Discovery: Evaluated using CIFAR-100 and CUB under continual category discovery protocols.

Key Insights and Ideas

  • The Complexity Advantage: Mathematically, the authors prove that restricting a model's focus to a smaller, predictive subset of size $s$ (out of $P$ total patches) yields a tighter empirical Rademacher complexity bound. This effectively reduces the model's capacity to overfit to background shortcuts.
  • Sparsity Flexibility: Unlike traditional token-pruning methods that discard a fixed percentage of patches for computational efficiency, PatchGen allows the mask size to adapt dynamically. If an entire image is relevant (e.g., a dense tumor sample), it can select all patches.
  • Independence from Text: PatchGen achieves performance competitive with vision-language models without requiring any text supervision, making it uniquely viable for open-world or unsupervised category discovery.

Key Results

PatchGen consistently outperformed matched-backbone baselines across all evaluated shift settings:

  • Natural Images: Paired with a DINOv2 ViT-L/14 backbone, PatchGen achieved a state-of-the-art average accuracy of 79.2% (and 80.9% when combined with SWAD) on DomainBed benchmarks.
  • Histopathology: On the newly introduced HISTOCOLON benchmark, PatchGen delivered clear performance gains, successfully isolating tumor-specific morphology from confounding inflammatory tissue.
Figure 1: Qualitative histopathology visualization on an unseen-domain tumor image: PatchGen’s learned soft predictive-subset mask compared with the corresponding attention maps of previous methods.
  • Category Discovery: In continual category discovery (CCD) tasks, PatchGen yielded its largest improvements on unknown/novel classes, demonstrating that isolating core visual evidence inherently aids in identifying entirely new categories.

Limitations and Future Work

Because the true "oracle" predictive subset within an image is fundamentally unobserved, PatchGen's training objectives cannot mathematically guarantee that the learned soft mask perfectly uncovers the true underlying biological or semantic boundaries. Additionally, the authors note that for vision-language backbones already heavily pre-trained to focus on semantic regions, the margin for improvement via additional patch selection may be narrower. Future work could explore tighter integration with self-supervised pre-training objectives to learn these masks even earlier in the training pipeline.

Summary updated

Abstract

Visual classifiers are expected to generalize under data shifts, target shifts, and their combinations, yet most existing methods focus on domain invariance while failing to address intra-image predictive sufficiency. We investigate the structural hypothesis that each image contains a sample-adaptive oracle intra-image predictive subset sufficient for label prediction, while the remaining patches form non-essential complementary context that may correlate with the label. The theoretical analysis shows that restricting prediction to this oracle subset preserves the Bayes risk achievable by the full-patch representation while admitting a complexity bound that tightens with the oracle-subset size. Based on this view, we propose PatchGen, a text-free module that learns a sample-dependent soft predictive-subset mask as a task-driven proxy for the unobserved oracle subset mask. Specifically, histopathology visualizations suggest that PatchGen assigns higher scores to tumor-consistent regions than to some frequently co-occurring inflammatory context. Extensive experiments on natural and histopathological image benchmarks spanning all three shift settings show that PatchGen improves average performance over matched-backbone baselines in most evaluated configurations, enhances generalization to unknown classes, and remains competitive with vision-language methods without text supervision.
BibTeX
@misc{tan2026patchgen,
  author = {Zhaorui Tan and Weimiao Yu and Xi Yang},
  title = {{PatchGen: Learning Soft Intra-Image Predictive Subsets for Visual Generalization}},
  year = {2026},
  eprint = {2608.12766},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.12766},
}
Figure