Transfer Representation Learning with TSK Fuzzy System

Peng Xu, Zhaohong Deng, Jun Wang, Qun Zhang, Shitong Wang

Abstract

Transfer learning leverages labeled data from a source domain to address tasks in an unlabeled target domain. A central challenge is learning a shared feature space where data distributions from both domains match, a process known as Transfer Representation Learning (TRL). Conventional TRL methods often rely on kernel-based nonlinear mappings, which lack interpretability and are difficult to select. This paper proposes TRL-TSK-FS, a framework that integrates the Takagi-Sugeno-Kang fuzzy system (TSK-FS) into transfer learning. By using the antecedent part of the TSK-FS for nonlinear mapping and the consequent part for linear transformation, the model creates an intuitive, interpretable feature space. Experimental results on text and image datasets demonstrate that TRL-TSK-FS significantly outperforms traditional kernel-based methods.

The Interpretability Gap in Transfer Learning

In transfer learning, finding a shared feature space is critical for bridging the gap between a labeled "source domain" and an unlabeled "target domain." Traditionally, researchers have relied on kernel methods to perform the nonlinear mappings required to align these distributions. However, kernels are often treated as mathematical "black boxes," offering little insight into the transformation process. Furthermore, selecting the optimal kernel (e.g., RBF vs. Linear) typically requires exhaustive trial and error.

The TRL-TSK-FS Framework

To address these limitations, the proposed TRL-TSK-FS replaces opaque kernels with interpretable fuzzy logic. The architecture decomposes the TSK fuzzy system into two distinct functional roles:

  1. Nonlinear Mapping (Antecedent Part): The "IF-THEN" rules of the fuzzy system map raw data into a high-dimensional fuzzy feature space. Unlike standard fuzzy clustering, which can be unstable, this method uses a deterministic algorithm called Var-Part to ensure consistent parameter initialization.
  2. Dimensionality Reduction (Consequent Part): This component acts as a linear projection, mapping the high-dimensional fuzzy space down to a low-dimensional shared subspace where the source and target distributions are aligned.
The architecture of the proposed TRL-TSK-FS framework, illustrating the transformation from primal to fuzzy feature spaces.

Optimization Strategy

The model is trained by minimizing a multi-objective function that balances three key goals:

  • Distribution Matching: Utilizing Maximum Mean Discrepancy (MMD) to ensure the source and target data are statistically similar in the new space.
  • Discriminant Information: Incorporating Linear Discriminant Analysis (LDA) to maintain class separability.
  • Geometric Preservation: Using Principal Component Analysis (PCA) to retain the underlying structure of the data.

Experimental Evaluation

The model was evaluated on two major benchmarks to test its versatility across different data types:

  • Office-Caltech (Image): This dataset includes images from four distinct domains: Amazon, Webcam, DSLR, and Caltech-256. The authors conducted 12 transfer tasks to evaluate how well the model could adapt features across different photographic environments.
  • NG20 (Text): A subset of the 20 Newsgroups dataset, focusing on high-dimensional text vectors across categories like technology, recreation, science, and politics.
Sample images from the four domains of the Office-Caltech dataset used for cross-domain transfer tasks.

Key Results and Insights

TRL-TSK-FS consistently outperformed established transfer learning algorithms such as TCA, JDA, and GFK:

  • Superior Accuracy: On the Office-Caltech dataset, the method achieved an average accuracy of 54.29%, a significant improvement over the 46.33% achieved by the next best competitor (TJM). On the NG20 text dataset, it reached 89.47% accuracy.
  • Efficiency: The model demonstrated rapid convergence, typically reaching optimal performance within just 5 iterations of the joint distribution adaptation process.
  • Flexibility: Because TRL-TSK-FS is a representation learning method, the resulting "cleaned" features can be used with any standard classifier, such as 1-Nearest Neighbor.

Conclusion and Future Work

By treating fuzzy rules as "interpretable kernels," TRL-TSK-FS provides a transparent alternative to traditional black-box models. While highly effective for homogeneous transfer learning, future research will explore extending the framework to heterogeneous tasks (e.g., mapping text to images) and incorporating Locality Preserving Projections (LPP) to better maintain local data relationships during transformation.

Summary updated

Abstract

Transfer learning can address the learning tasks of unlabeled data in the target domain by leveraging plenty of labeled data from a different but related source domain. A core issue in transfer learning is to learn a shared feature space in where the distributions of the data from two domains are matched. This learning process can be named as transfer representation learning (TRL). The feature transformation methods are crucial to ensure the success of TRL. The most commonly used feature transformation method in TRL is kernel-based nonlinear mapping to the high-dimensional space followed by linear dimensionality reduction. But the kernel functions are lack of interpretability and are difficult to be selected. To this end, the TSK fuzzy system (TSK-FS) is combined with transfer learning and a more intuitive and interpretable modeling method, called transfer representation learning with TSK-FS (TRL-TSK-FS) is proposed in this paper. Specifically, TRL-TSK-FS realizes TRL from two aspects. On one hand, the data in the source and target domains are transformed into the fuzzy feature space in which the distribution distance of the data between two domains is min-imized. On the other hand, discriminant information and geo-metric properties of the data are preserved by linear discriminant analysis and principal component analysis. In addition, another advantage arises with the proposed method, that is, the nonlinear transformation is realized by constructing fuzzy mapping with the antecedent part of the TSK-FS instead of kernel functions which are difficult to be selected. Extensive experiments are conducted on the text and image datasets. The results obviously show the superiority of the proposed method.
BibTeX
@article{xu2020transfer,
  author = {Peng Xu and Zhaohong Deng and Jun Wang and Qun Zhang and Shitong Wang},
  title = {{Transfer Representation Learning with TSK Fuzzy System}},
  journal = {IEEE Transactions on Fuzzy Systems},
  year = {2020},
  doi = {10.1109/tfuzz.2019.2958299},
}
Figure