TDG: Text-guided Domain Generalization

Geng Liu, Yuxi WangarXiv 2023 · OpenReview

TL;DR

This paper introduces TDG, a novel approach to domain generalization that leverages text to improve model robustness across unseen domains. By generating domain-specific text descriptions and integrating them with image features, TDG significantly enhances generalization performance, demonstrating the power of multimodal information in tackling domain shift.

What Problem Does This Paper Address?

Traditional machine learning models often struggle when faced with data from domains different from their training data, a phenomenon known as "domain shift." This is particularly problematic in real-world scenarios where data distributions can vary significantly (e.g., photos vs. sketches, different lighting conditions, diverse camera angles). Domain generalization (DG) aims to build models that can perform well on unseen target domains without any further training or adaptation. While existing DG methods have shown promise, their performance often plateaus. This paper argues that Visual-Language Pre-trained (VLP) models, which excel at bridging vision and language, offer a powerful avenue for DG. The challenge is to effectively harness the rich semantic information present in text to guide the generalization process, especially since image and text features typically reside in different spaces.

Methods

The proposed TDG (Text-guided Domain Generalization) framework comprises three key components:

  1. Domain-Relevant Words Generation: To capture domain-specific nuances, the authors employ a pre-trained lexical substitution model. By providing a simple sentence like "an image of a dog" and instructing the model to substitute the word "image," they generate a diverse set of words (e.g., "photo," "painting," "sketch," "portrait") that describe various visual domains.
  2. Text Generation via Prompt Learning: These generated domain words are then combined with category words (e.g., "dog," "elephant") using a learnable text prompt template (e.g., "[domain] [category]"). This template is trained to produce text features that are aligned with image features of the same category but distinct across different domains. Two loss functions, $L_\alpha$ and $L_s$, are introduced: $L_\alpha$ encourages similarity between text and image features of the same category, while $L_s$ maximizes the difference between text features of the same category but different domains. This ensures text features retain category information while capturing domain diversity.
  3. Normalized Classifier: Leveraging the fact that VLP models often project images and text into a shared feature space where orientation is key, TDG employs a normalized classifier. Both input features and classifier parameters are normalized to unit vectors. This approach is more robust to variations in feature magnitudes, aligning better with the characteristics of VLP models like CLIP. The image encoder is also trained by backpropagating gradients from this classifier.

Datasets

The proposed TDG method was evaluated on four standard domain generalization benchmarks:

  • PACS: Contains images from four domains (Photo, Art, Cartoon, Sketch) across seven categories.
  • OfficeHome: A large-scale dataset with 65 classes across four domains (Art, Clipart, Product, Real-World).
  • VLCS: Features five classes collected from four different photo datasets, exhibiting domain shifts due to scene variations and viewpoints.
  • TerraIncognita: Includes photos of wild animals from four different locations, with variations in scene and object size.

The experiments utilized a "leave-one-domain-out" protocol, where one domain was held out as the target, and the remaining were used as source domains for training.

Key Insights and Ideas

  • Text as a powerful domain descriptor: The paper convincingly shows that text, when crafted to be domain-specific, can provide crucial information for generalization, complementing image-based features.
  • Domain diversity through text generation: The method effectively generates diverse textual descriptions that capture subtle domain differences, which is key to improving model robustness.
  • Shared feature space benefits: By aligning text and image features in a common space and using a normalized classifier, TDG effectively leverages the strengths of VLP models for DG.
  • Prompt learning for controlled text generation: The prompt learning approach allows for fine-grained control over the generated text, ensuring it carries both category and domain-specific information.

Key Results

TDG achieved state-of-the-art performance on several domain generalization benchmarks. For instance:

  • On PACS, TDG improved the baseline ERM performance by an average of 1.13% and achieved the best overall performance among the evaluated methods.
  • On OfficeHome, TDG showed an average improvement of 1.83% over the baseline and outperformed existing methods like MIRO, CoOp, and CoCoOp.
  • On VLCS and TerraIncognita, TDG also demonstrated significant improvements, highlighting its effectiveness even when domain shifts are primarily due to image variations rather than stylistic differences.
  • Ablation studies confirmed the effectiveness of both the text generation module and the normalized classifier, with the latter showing particular benefit when used with CLIP-based models.

Limitations and Future Work

  • Computational Cost: While the paper emphasizes easy implementation, generating and processing text features adds computational overhead compared to purely image-based methods.
  • Reliance on VLP Models: The success of TDG is heavily dependent on the quality and capabilities of the underlying VLP model (e.g., CLIP).
  • Future Work: The authors suggest exploring more sophisticated text generation strategies and investigating how TDG can be applied to other vision tasks beyond classification. Further research into optimizing the prompt learning process and exploring different VLP architectures could also yield improvements.

Summary updated

Abstract

Domain generalization (DG) attempts to generalize a model trained on single or multiple source domains to the unseen target domain. Benefiting from the success of Visual-and-Language Pre-trained models in recent years, we argue that it is crucial for domain generalization by introducing extra text information. In this paper, we develop a novel Text-guided Domain Generalization (TDG) paradigm for domain generalization, which includes three following aspects. Specifically, we first devise an automatic words generation method to extend the description of current domains with novel domain-relevant words. Then, we embed the generated domain information into the text feature space, by the proposed prompt learning-based text feature generation method, which shares a common representation space with the image feature. Finally, we utilize both input image features and generated text features to train a specially designed classifier that generalizes well on unseen target domains, while the image encoder is also updated under the supervision of gradients back propagated from the classifier. Our experimental results show that the techniques incorporated by TDG contribute to the performance in an easy implementation manner. Experimental results on several domain generalization benchmarks show that our proposed framework achieves superior performance by effectively leveraging generated text information in domain generalization.
BibTeX
@misc{liu2023tdg,
  author = {Geng Liu and Yuxi Wang},
  title = {{TDG: Text-guided Domain Generalization}},
  year = {2023},
  eprint = {2308.09931},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.09931},
}
Figure