TL;DR
This paper introduces a novel distributed method for collaborative feature selection that preserves data privacy. It allows multiple parties to jointly select the most discriminative features without sharing their original data, by leveraging intermediate representations and an embedded feature selection approach. This is particularly useful in scenarios where data is siloed due to privacy concerns but a comprehensive feature selection is needed.
What Problem Does This Paper Address?
High-dimensional data presents challenges like the "curse of dimensionality," making it difficult for machine learning models to perform well. Feature selection is a technique to address this by identifying and keeping only the most relevant features. However, many advanced feature selection methods require access to large amounts of data to learn effectively. In real-world scenarios, data is often distributed across multiple parties (e.g., different hospitals, organizations), and sharing this data is restricted due to privacy concerns. This data distribution can lead to information deficiency in individual parties, hindering their ability to perform robust feature selection. Existing privacy-preserving methods often focus on filter or wrapper approaches, leaving a gap for distributed embedded feature selection, which integrates feature selection into the model training process.
Methods
The core of the proposed method is a distributed, privacy-preserving approach to embedded feature selection. It builds upon the idea of transforming data into a low-dimensional space while preserving local data structure, a common theme in embedded methods.
Here's a breakdown of the key techniques:
- Intermediate Representations: Instead of sharing raw data, each party computes an "intermediate representation" of its data using a transformation matrix ($B_i$). This representation, along with the transformation matrix, is kept private within the party.
- Anchor Data: To enable collaboration, a set of "anchor data" is introduced. This anchor data is shared among all parties. Each party computes an intermediate representation of this anchor data.
- Collaborative Transformation: The intermediate representations of the anchor data from all parties are then used to compute a common low-dimensional representation ($Z^\alpha$). This allows for the calculation of a global transformation matrix ($M_i$) for each party, which projects the data into a shared low-dimensional space.
- Embedded Feature Selection: The method then adapts an existing embedded feature selection framework. For each party, it learns a transformation matrix ($W_i = (M_i B_i)^T$) that projects the original data into the shared low-dimensional space. This matrix $W_i$ is optimized to simultaneously:
- Reconstruct the transformed data ($Z_i = W_i^T X_i$).
- Encourage row sparsity in $W_i$ (using an $L_{2,1}$-norm penalty), which directly leads to feature ranking.
- Preserve the local data structure of the original data ($X_i$).
- Distributed Optimization: The optimization problem is solved iteratively by updating $W_i$ and the intermediate representations ($X_i$) in each party. The algorithm is designed to converge to a stable solution.
What's novel is the combination of these elements: using intermediate representations for privacy, anchor data for alignment, and adapting embedded feature selection for a distributed, privacy-preserving setting.
Datasets
The paper evaluates the proposed method on two types of datasets:
- Handwritten Digit Data:
- MNIST-d1: Contains digits 0, 1, 2, 3. The dataset is artificially split into two parties with different distributions of these digits.
- MNIST-d2: Contains digits 4, 5, 6, 7, 8, 9. This dataset is split into six parts, with each part having a skewed distribution of digits.
- Gene Expression Data:
- Colon: Downloaded from http://featureselection.asu.edu/datasets.php.
- TOX-171: Downloaded from http://featureselection.asu.edu/datasets.php.
- Lung: Downloaded from http://featureselection.asu.edu/datasets.php.
- Lymphoma: Downloaded from http://featureselection.asu.edu/datasets.php. These datasets are randomly divided into two or three parts, with 80% of the data used for training and 20% for testing in each party.
Key Insights and Ideas
- Privacy-Preserving Collaboration is Achievable for Embedded Methods: The paper demonstrates that it's possible to perform sophisticated embedded feature selection in a distributed manner without compromising data privacy, a significant step beyond existing filter/wrapper privacy-preserving methods.
- Intermediate Representations as a Privacy Shield: By sharing transformed representations of data (and specifically, anchor data) rather than raw data, the method effectively decouples parties while still allowing for collaborative learning.
- Anchor Data Bridges the Gap: The use of anchor data is crucial for aligning the different parties' data representations in a common low-dimensional space, enabling meaningful collaboration.
- Embedded Methods Benefit from Distributed Data: Just as centralized embedded methods can overcome local information deficiency, this distributed approach allows parties to leverage each other's data (indirectly) to learn more discriminative features than they could individually.
- Trade-off Between Local and Global Structure: The method balances preserving the local data structure within a party with learning a global structure across parties, which can lead to better performance than purely local or purely global approaches in certain scenarios.
Key Results
The experimental results show that the proposed Distributed Collaborative Feature Selection (DCFS) method generally outperforms Local Feature Selection (LFS) within individual parties, and often performs comparably to or better than collecting all data for centralized analysis (AFS), especially when data is unevenly distributed.
- Handwritten Digits (MNIST):
- On MNIST-d1, DCFS achieved the best results in Party 2, outperforming LFS and even AFS in terms of accuracy (96.7% vs. 97.2% for AFS, with a standard deviation of 1.0 vs. 0.5).
- On MNIST-d2, DCFS consistently performed well, often matching or slightly exceeding AFS across different parties.
- Qualitative results (Figure 1) show that features selected by DCFS better capture data structure, leading to more recognizable digits compared to LFS.
- Gene Expression Data:
- DCFS showed significant improvements over LFS and often performed better than AFS, particularly in terms of NMI (Normalized Mutual Information). For example, on the TOX-171 dataset, Party 1's NMI for DCFS was 46.1% compared to 43.7% for AFS.
- The paper notes that DCFS can be seen as a trade-off that sometimes yields better results than AFS, which might over-emphasize global structure preservation at the expense of local nuances.
- Figure 2 demonstrates that DCFS generally achieves higher accuracy and NMI than LFS as the number of selected features increases.
Limitations and Future Work
- Privacy Guarantees: The authors acknowledge that the method does not employ formal privacy-preserving techniques like cryptography or differential privacy. The privacy is based on the assumption that original data cannot be approximated without both the intermediate representation and the transformation matrix, which are kept private. They suggest further formal analysis of privacy is needed.
- Anchor Data Selection: The current anchor data generation is based on random values within the range of training data. The paper suggests investigating more sophisticated methods for selecting anchor data to potentially improve performance.
- Supervised Case and Algorithm Acceleration: The authors plan to extend the method to the supervised case and explore techniques to accelerate the algorithm.
- Statistical Datasets: The current privacy analysis does not consider the privacy of statistical datasets.
- Parameter Tuning: While experiments show parameter sensitivity is within a reasonable range, optimal parameter selection for $\alpha$ and $\beta$ might still require tuning.
Summary updated