Self-Disentanglement and Re-Composition for Cross-Domain Few-Shot Segmentation

Jintao Tong, Yixiong Zou, Guangyao Chen, Yuhua Li, Ruixuan Li

The Challenge of Cross-Domain Few-Shot Segmentation

Cross-Domain Few-Shot Segmentation (CD-FSS) represents one of the most demanding frontiers in computer vision. The goal is to train a model on a massive, easily accessible dataset (such as natural images of everyday objects) and then task it with segmenting completely novel classes in entirely different domains (such as medical X-rays or high-resolution satellite imagery) using only one or five labeled examples.

The standard approach to this problem involves computing the "distance" between the features of the labeled example (the support) and the new image (the query). However, researchers have identified a critical flaw in this widely adopted method when using Vision Transformers (ViTs): feature entanglement.

The Root Cause: Feature Entanglement in Vision Transformers

In a standard Vision Transformer, the final output is a cumulative sum of many individual layers. When the model compares a support image to a query image, it is essentially comparing every pattern in the first image with every pattern in the second simultaneously.

This aggregation leads to "meaningless comparisons." For instance, if a model entangles the patterns of a bird's wings with its body, it may fail to recognize a different species in a new domain that shares similar wings but has a different body type. Because the model treats rational matches (wings-to-wings) and noisy matches (wings-to-bodies) with equal importance, its ability to generalize across domains is severely compromised.

The SDRC Framework: Disentangle and Re-Compose

To overcome this bottleneck, researchers propose a novel framework called Self-Disentanglement and Re-Composition (SDRC). The architecture addresses feature entanglement through three primary mechanisms:

  1. Structural Decomposition: Leveraging the residual nature of ViTs, the SDRC framework treats the final output as a sum of individual layers. The authors demonstrate that these individual layers naturally capture distinct semantic parts of an image, such as outlines in one layer and textures in another.
  2. Orthogonal Space Decoupling (OSD): To prevent these distinct patterns from bleeding into one another, the OSD module forces the features from different layers into an orthogonal space. By minimizing the mutual information between layers, the model ensures that each layer represents a truly unique, disentangled pattern.
  3. Cross-Pattern Comparison (CPC) & Adaptive Fusion Weight (AFW): Instead of relying on a single global comparison, the CPC module cross-matches all layers against each other. During the fine-tuning phase on the target domain, the AFW learns which of these layer-to-layer comparisons are "rational" for the new domain. It effectively re-composes the features by weighing the meaningful matches and ignoring the noise.

Experimental Setup and Key Results

The SDRC model was trained on the PASCAL VOC 2012 dataset (natural images) and evaluated across four highly diverse target domains: FSS-1000 (natural images), DeepGlobe (satellite imagery), ISIC2018 (skin lesion medical images), and Chest X-ray (tuberculosis screening radiographs).

The framework established a new state-of-the-art for CD-FSS, outperforming previous top methods by an average of 1.92% in 1-shot and 1.88% in 5-shot settings across all benchmarks. The gains were particularly striking in difficult medical domains; for example, SDRC achieved an impressive 82.86% mIoU on the Chest X-ray dataset, significantly surpassing more complex baseline models.

Insights, Efficiency, and Future Directions

One of the most surprising insights from this research is that layer-wise similarity between domains is often higher than the similarity of the final aggregated output. This suggests that the standard "summing" process in Transformers is the very mechanism creating the entanglement that hinders transfer learning. By allowing the model to dynamically weigh cross-layer matches, it successfully ignores domain-specific noise and focuses on domain-invariant patterns.

Furthermore, SDRC achieves these results with remarkable efficiency. Unlike many recent state-of-the-art methods that rely on heavy adapter networks or complex decoders, SDRC utilizes an encoder-only structure. It delivers superior performance with fewer parameters and lower computational cost (FLOPs) simply by optimizing how existing ViT layers are utilized.

While the results across the four target domains are robust, future work will focus on evaluating the model across an even wider breadth of real-world scenarios. Additionally, researchers see a promising path in connecting this framework with causal reasoning to better interpret exactly why certain layer matches prove more rational than others during cross-domain transfer.

Summary updated

Abstract

Cross-Domain Few-Shot Segmentation (CD-FSS) aims to transfer knowledge from a large-scale source-domain dataset to unseen target-domain datasets with limited annotated samples. Current methods typically compare the distance between training and testing samples for mask prediction. However, a problem of feature entanglement exists in this well-adopted method, which binds multiple patterns together and harms the transferability. However, we find an entanglement problem exists in this widely adopted method, which tends to bind source-domain patterns together and make each of them hard to transfer. In this paper, we aim to address this problem for the CD-FSS task. We first find a natural decomposition of the ViT structure, based on which we delve into the entanglement problem for an interpretation. We find the decomposed ViT components are crossly compared between images in distance calculation, where the rational comparisons are entangled with those meaningless ones by their equal importance, leading to the entanglement problem. Based on this interpretation, we further propose to address the entanglement problem by learning to weigh for all comparisons of ViT components, which learn disentangled features and re-compose them for the CD-FSS task, benefiting both the generalization and finetuning. Experiments show that our model outperforms the state-of-the-art CD-FSS method by 1.92% and 1.88% in average accuracy under 1-shot and 5-shot settings, respectively.
BibTeX
@inproceedings{tong2025self,
  author = {Jintao Tong and Yixiong Zou and Guangyao Chen and Yuhua Li and Ruixuan Li},
  title = {{Self-Disentanglement and Re-Composition for Cross-Domain Few-Shot Segmentation}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=LkgwdSzlb6},
}
Figure