Dynamically Decoding Source Domain Knowledge For Unseen Domain Generalization

Cuicui Kang, Karthik Nandakumar

TL;DR

This paper introduces D2SDK, a novel Transformer-based approach for domain generalization (DG) that dynamically decodes knowledge from multiple source domains during inference. By leveraging a Transformer encoder to consolidate source domain knowledge and a decoder to selectively retrieve relevant information based on input similarity, D2SDK significantly improves generalization to unseen domains, outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

In real-world applications, machine learning models often encounter data from domains they weren't trained on, a phenomenon known as "domain shift." Domain generalization (DG) aims to build models that can perform well on these unseen target domains without requiring any data from them during training. While many methods focus on learning domain-invariant features or using data augmentation, existing "mixture of experts" (MoE) approaches, which combine specialized models for each source domain, often fail to fully utilize the rich knowledge within these source domains during inference. They typically rely on simple weighted combinations, which can be suboptimal when a target domain is a complex blend of multiple source domains or when specific features from different source domains are more relevant. This paper addresses the limitation of MoE methods by proposing a more sophisticated mechanism to dynamically leverage source domain knowledge.

Methods

The core of the proposed method, D2SDK (Dynamically Decoding Source Domain Knowledge), is a hybrid architecture that combines domain-specific "local experts" with a Transformer-based "cross-domain Transformer."

Here's a breakdown:

  • Shared Backbone: A standard Convolutional Neural Network (CNN) backbone (e.g., ResNet) is used to extract generic low-level features from the input image. This backbone is shared across all components to save computation and memory.
  • Local Experts: For each source domain, a domain-specific expert is created. This expert consists of a convolutional neck and a classifier, trained to learn discriminative features for its respective source domain.
  • Domain-Agnostic Query Branch: A separate branch also uses the shared backbone to extract features that will serve as a "query" for the Transformer decoder.
  • Cross-Domain Transformer: This is the novel component.
  • Encoder: The features from all domain-specific local experts are fed into a Transformer encoder. This encoder learns dependencies and consolidates the knowledge from all source domains into a "memory" representation.
  • Decoder: The domain-agnostic query features interact with the encoder's memory through a cross-attention mechanism. This allows the decoder to dynamically select and weigh information from different source domains based on their similarity to the current input. Domains that are more similar to the input will contribute more to the final representation.
  • Classification: The output of the Transformer decoder is then passed through a final fully connected layer for classification.

This Transformer-based decoding mechanism allows for a more nuanced and adaptive utilization of source domain knowledge compared to simple weighted averaging.

Datasets

The proposed D2SDK method was evaluated on three standard domain generalization benchmarks:

  • PACS: Contains images from four domains: Photo, Art Painting, Cartoon, and Sketches.
  • Office-Home: A larger dataset with 65 object categories across four domains: Art, Clipart, Product, and Real-World.
  • VLCS: Comprises images from five object categories shared across four domains: Caltech, Labelme, Pascal, and Sun.

The experimental protocol used was the standard leave-one-domain-out cross-validation, where three domains are used for training and one is held out as the target domain.

Key Insights and Ideas

  • Dynamic Knowledge Decoding: The most significant insight is that simply combining source domain features with fixed weights is insufficient. The proposed Transformer decoder can dynamically "decode" relevant knowledge from source domains based on the input's characteristics, leading to better generalization.
  • Transformer for Cross-Domain Interaction: Adapting Transformers, particularly their cross-attention mechanism, is highly effective for modeling relationships between different source domains and selectively leveraging them for unseen data.
  • Source Domain Similarity Matters: The cross-attention mechanism inherently prioritizes source domains that are more similar to the test input, mimicking human intuition when facing novel situations.
  • MoE Limitations Addressed: The paper effectively tackles the limitations of traditional MoE methods by moving beyond simple aggregation to a more intelligent, context-aware knowledge retrieval process.

Key Results

The D2SDK method demonstrated state-of-the-art performance across the evaluated datasets:

  • PACS: Achieved the best average performance and the best performance on the Cartoon target domain, outperforming the second-best method (DAEL) by a clear margin.
  • Office-Home: Outperformed existing state-of-the-art methods on average and achieved the best performance on two of the four target domains. This is particularly noteworthy given the dataset's large number of categories (65), making it a more challenging benchmark.
  • VLCS: While D2SDK achieved the best performance, the authors note a potential unfair comparison due to the use of a ResNet-18 backbone versus AlexNet in some other benchmarked methods.

The ablation studies also confirmed the importance of the Transformer decoder, showing that removing it significantly degraded performance. The method was also found to be relatively insensitive to the weighting parameter $\lambda$ for the domain expert loss.

Limitations and Future Work

  • Computational Complexity: The authors acknowledge that the proposed method has higher computational complexity during inference compared to baseline architectures. They suggest that discarding the Transformer encoder or even the self-attention block in the decoder could be avenues for reducing this cost, though further study is needed.
  • Transformer Architecture Complexity: Future work could focus on simplifying the Transformer architecture to reduce its complexity while maintaining or enhancing its ability to model diverse domain shifts.
  • Model Selection: The paper discusses the practice of "test-set best-epoch performance" and strongly discourages it, advocating for results based on the last epoch of training for a fairer comparison. This highlights a potential issue in how DG performance is reported in some literature.

Summary updated

Abstract

Optimizing the performance of classifiers on samples from unseen domains remains a challenging problem. While most existing studies on domain generalization focus on learning domain-invariant feature representations, multi-expert frameworks have been proposed as a possible solution and have demonstrated promising performance. However, current multi-expert learning frameworks fail to fully exploit source domain knowledge during inference, resulting in sub-optimal performance. In this work, we propose to adapt Transformers for the purpose of dynamically decoding source domain knowledge for domain generalization. Specifically, we build one domain-specific local expert per source domain and one domain-agnostic feature branch as query. A Transformer encoder encodes all domain-specific features as source domain knowledge in memory. In the Transformer decoder, the domain-agnostic query interacts with the memory in the cross-attention module, and domains that are similar to the input will contribute more to the attention output. Thus, source domain knowledge gets dynamically decoded for inference of the current input from unseen domain. This mechanism enables the proposed method to generalize well to unseen domains. The proposed method has been evaluated on three benchmarks in the domain generalization field and shown to have the best performance compared to state-of-the-art methods.
BibTeX
@misc{kang2021dynamically,
  author = {Cuicui Kang and Karthik Nandakumar},
  title = {{Dynamically Decoding Source Domain Knowledge For Unseen Domain Generalization}},
  year = {2021},
  eprint = {2110.03027},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.03027},
}
Figure