Transfer Representation Learning With TSK Fuzzy System

Peng Xu, Zhaohong Deng, Jun Wang, Qun Zhang, Kup‐Sze Choi, Shitong Wang

Abstract

Transfer learning addresses the challenge of unlabeled data in a target domain by leveraging labeled data from a related source domain. A fundamental goal in this field is Transfer Representation Learning (TRL), which seeks to map both domains into a shared feature space where their distributions are aligned. While kernel-based nonlinear mappings are commonly used for this transformation, they often function as "black boxes" that lack interpretability and require difficult parameter selection. This paper proposes TRL-TSK-FS, a novel framework that replaces traditional kernel functions with a Takagi-Sugeno-Kang fuzzy system (TSK-FS). By utilizing the antecedent part of the TSK-FS for nonlinear mapping, the model achieves high performance while maintaining linguistic interpretability. The framework further incorporates distribution alignment and structural preservation to ensure robust knowledge transfer. Experimental results on image and text datasets demonstrate that TRL-TSK-FS significantly outperforms state-of-the-art transfer learning methods.

Introduction

In traditional machine learning, models are typically trained and tested on data originating from the same distribution. However, real-world applications often involve a "distribution shift," where the source data (labeled) and target data (unlabeled) differ significantly. Transfer Learning bridges this gap by finding a common representation that minimizes the discrepancy between these domains.

Most existing TRL methods rely on kernel-based nonlinear mapping followed by linear dimensionality reduction. Despite their effectiveness, kernels are mathematically opaque and their selection is often a matter of trial and error. This research introduces an interpretable alternative using TSK fuzzy systems, which provide a rule-based approach to feature transformation that is both intuitive and mathematically rigorous.

Methodology: The TRL-TSK-FS Framework

The proposed TRL-TSK-FS model realizes transfer representation learning through a two-stage process integrated into a single optimization framework.

1. Nonlinear Fuzzy Mapping

Instead of using a kernel function, the model uses the antecedent (IF) part of TSK fuzzy rules to map raw data into a high-dimensional fuzzy feature space. To ensure the model is stable and repeatable, the authors utilize a deterministic clustering algorithm called Var-Part rather than the standard, randomized Fuzzy C-Means. This ensures that the fuzzy membership functions are consistent across different runs.

2. Linear Dimensionality Reduction and Alignment

The consequent (THEN) part of the fuzzy rules acts as a linear projection, mapping the high-dimensional fuzzy data into a lower-dimensional shared space. The model optimizes this projection by balancing three critical objectives:

  • Distribution Alignment: Using Maximum Mean Discrepancy (MMD) to minimize the distance between the source and target distributions.
  • Structure Preservation: Integrating logic from Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) to preserve the geometric properties and class-separability of the data.
  • Iterative Pseudo-Labeling: Since the target domain lacks labels, the model iteratively predicts labels for the target data to refine the alignment process.

Key Insights: Interpretability and Stability

A major contribution of this work is the shift from "black-box" kernels to interpretable fuzzy rules. In TRL-TSK-FS, the feature transformation can be understood through linguistic labels (e.g., "Low," "Middle," "High"). A researcher can inspect a rule and understand that "If Feature A is High and Feature B is Low, then the data belongs to a specific region of the shared space."

Furthermore, the use of the Var-Part algorithm addresses a common criticism of fuzzy systems: instability. By making the rule generation process deterministic, the authors ensure that the learned representation is reliable for practical deployment.

Experimental Results

The model was evaluated against several benchmark datasets, including the Office-Caltech image collection and the NG20 text dataset. The results indicate that TRL-TSK-FS is highly effective even with a small number of fuzzy rules (e.g., $K=3$).

  • Image Classification: On the Office-Caltech dataset, TRL-TSK-FS achieved an average accuracy of 54.29%, outperforming the next best method (TJM) by nearly 8% and the standard baseline by 23%.
  • Text Classification: On the NG20 dataset, the model reached an accuracy of 89.47%, surpassing established methods like Joint Distribution Adaptation (JDA).

Conclusion and Future Work

TRL-TSK-FS demonstrates that fuzzy systems can serve as a powerful and interpretable alternative to kernel methods in transfer learning. By combining the nonlinear mapping capabilities of fuzzy antecedents with the structural preservation of PCA and LDA, the model provides a robust solution for cross-domain data alignment.

Future research will focus on extending this framework to heterogeneous transfer learning, where the source and target domains have different initial feature dimensions. Additionally, the authors aim to incorporate Locality Preserving Projections (LPP) to better capture local manifold structures and improve scalability for large-scale datasets.

Summary updated

Abstract

Transfer learning can address the learning tasks of unlabeled data in the target domain by leveraging plenty of labeled data from a different but related source domain. A core issue in transfer learning is to learn a shared feature space where the distributions of the data from the two domains are matched. This learning process can be named as transfer representation learning (TRL). Feature transformation methods are crucial to ensure the success of TRL. The most commonly used feature transformation method in TRL is kernel-based nonlinear mapping to the high-dimensional space, followed by linear dimensionality reduction. But the kernel functions are lack of interpretability, and it is difficult to select kernel functions. To this end, this article proposes a more intuitive and interpretable method, called TRL with TSK-FS (TRL-TSK-FS), by combining TSK fuzzy system (TSK-FS) with transfer learning. Specifically, TRL-TSK-FS realizes TRL from two aspects. On one hand, the data in the source and target domains are transformed into the fuzzy feature space where the distribution distance of the data between the two domains is minimized. On the other hand, discriminant information and geometric properties of the data are preserved by linear discriminant analysis and principal component analysis. A further advantage is that nonlinear transformation is realized in the proposed method by constructing fuzzy mapping with the antecedent part of the TSK-FS instead of kernel functions, which are difficult to be selected. Extensive experiments are conducted on text and image datasets to demonstrate the superiority of the proposed method.
BibTeX
@article{xu2020transfer,
  author = {Peng Xu and Zhaohong Deng and Jun Wang and Qun Zhang and Kup‐Sze Choi and Shitong Wang},
  title = {{Transfer Representation Learning With TSK Fuzzy System}},
  journal = {IEEE Transactions on Fuzzy Systems},
  year = {2020},
  doi = {10.1109/tfuzz.2019.2958299},
}
Figure