Internal Robust Representations for Domain Generalization

Mohammad RostamiAAAI 2023 · DOI

Introduction

The standard paradigm of machine learning assumes that training and testing data are independent and identically distributed (i.i.d.). However, in practical applications, this assumption is frequently violated. Whether through temporal drift—where data characteristics evolve over time—or domain shift—where a model trained in one environment is deployed in another—distribution shifts often lead to significant performance degradation. This paper explores a unified strategy to mitigate these issues by "consolidating" the internal representations within a neural network, ensuring consistency across varying domains and sequential tasks.

The Challenge of Representation Alignment

When a predictive model $f_\theta$ is decomposed into an encoder $e_w(\cdot)$ and a classifier $c_v(\cdot)$, the failure to generalize can often be traced back to the encoder. While the classifier learns a decision boundary based on the features it receives during training, a shift in the input domain causes the encoder to produce feature clusters that no longer align with that boundary.

This misalignment manifests in two primary research areas:

  1. Unsupervised Domain Adaptation (UDA): The challenge of adapting a model to a new, unlabeled target domain.
  2. Continual Learning (CL): The challenge of learning new tasks sequentially without "catastrophic forgetting" of previous knowledge.

Proposed Methodology: Internal Consolidation

The author proposes a parametric approach to model decomposition, treating the internal representation space as the critical junction for generalization.

Unsupervised Domain Adaptation

In UDA, the goal is to ensure that the target data "looks" like the source data from the perspective of the classifier. This is achieved by minimizing a probability metric $D(\cdot, \cdot)$—such as the Wasserstein distance or Sliced Wasserstein Distance—between the internal representations of the source and target domains. By adapting the encoder to align these distributions, the pre-trained classifier can be utilized effectively on the target domain without requiring target labels.

Continual Learning

To address catastrophic forgetting in CL, the author suggests modeling the internal distribution of the initial task parametrically (e.g., using Gaussian Mixture Models). When the model encounters subsequent tasks, it is trained under a constraint: the internal representation space must remain stable and shared. This ensures that the features extracted for a new task do not distort the feature space required for previous tasks, allowing the model to maintain a consistent "internal language" across its lifetime.

Empirical Applications

The efficacy of this consolidation strategy has been demonstrated across several domains in research conducted between 2020 and 2023:

  • Semantic Segmentation: Maintaining accuracy in urban environment mapping across different cities or weather conditions.
  • Medical Imaging: Facilitating "source-free" adaptation, where privacy concerns prevent access to the original training data, requiring the model to adapt using only its internal logic and new, unlabeled images.
  • General Classification: Improving robustness on standard benchmarks for lifelong learning and domain adaptation.

Key Insights

The central thesis of this work is that the encoder, rather than the classifier logic, is the primary bottleneck for generalization. By forcing the model to maintain a robust, shared feature space—a process termed "internal consolidation"—researchers can bridge the gap between spatial/contextual shifts (DA) and temporal shifts (CL). This unified framework suggests that these two fields are fundamentally solving the same problem of distribution management.

Limitations and Future Directions

Despite its success, the approach faces certain hurdles. The choice of the probability metric $D$ is non-trivial, as different metrics may be better suited to specific data geometries. Furthermore, the reliance on parametric assumptions (e.g., assuming a specific functional form for the internal distribution) may limit the model's flexibility when dealing with highly complex or non-standard data distributions.

Future research may extend this consolidation strategy to Reinforcement Learning or Out-of-Distribution (OOD) detection, potentially establishing it as a general-purpose strategy for robust machine learning.

Summary updated

BibTeX
@inproceedings{rostami2023internal,
  author = {Mohammad Rostami},
  title = {{Internal Robust Representations for Domain Generalization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.26818},
}
Figure