TL;DR
AdaMuS introduces an adaptive multi-view representation learning framework specifically designed for dimensionally unbalanced data (e.g., combining $10^6$-dimensional video frames with $10^1$-dimensional physiological signals). By combining a parameter-free encoder pruning method with a sparse batch normalization fusion paradigm, it prevents low-dimensional views from overfitting while successfully filtering out redundancy and alignment noise.
What Problem Does This Paper Address?
In multi-view learning, algorithms fuse diverse feature sets (or "views") to build a comprehensive representation of data. Most existing frameworks operate under the implicit assumption that these views share similar dimensionalities. However, real-world applications frequently violate this assumption. For instance, in emotion recognition, video data can exceed $10^6$ dimensions, while accompanying heart rate or galvanic skin responses might only consist of $10^1$ dimensions.
When dealing with such extreme dimensional disparities, standard models suffer from two major drawbacks:
- The Dominance Issue: Directly concatenating views causes the high-dimensional features to overwhelm the low-dimensional ones, causing the model to degenerate into single-view learning.
- Overfitting and Redundancy: Forcing low-dimensional views into a high-dimensional space using standard Deep Neural Networks (DNNs) introduces massive parameter redundancy. This mismatch leads to severe overfitting and forces the model to generate meaningless "pseudo-dimensions" to satisfy alignment constraints.
Methods
The authors propose AdaMuS (Adaptive Multi-view Sparsity Learning) to bridge this dimensional chasm using three core components:
- Principal Neuron Analysis (PNA): To map low-dimensional views into a unified space without overfitting, AdaMuS uses view-specific encoders optimized by PNA. Inspired by Principal Component Analysis (PCA), PNA calculates a unique, parameter-free pruning rate for each layer by analyzing neuron correlations and the view's specific imbalance degree. It then removes highly correlated, redundant neurons in a one-shot pruning step.
- Multi-view Sparse Batch Normalization (MSBN): During feature fusion, instead of assigning a global weight to an entire view, MSBN penalizes individual latent dimensions. By applying an $\ell_1$-norm constraint to the scaling factors ($\gamma$) of the batch normalization layer, the model automatically suppresses redundant dimensions generated purely for forced alignment.
- Self-Supervised Contrastive Learning: To train the network without relying on manual labels, AdaMuS constructs a balanced K-nearest neighbor (KNN) similarity graph for each view. These are combined into a unified consensus matrix to generate binary pseudo-labels, guiding a contrastive loss function that pulls similar instances together and pushes distinct ones apart.
Datasets
The framework was evaluated on a synthetic toy dataset and seven real-world benchmarks:
- Synthetic Toy Dataset: 1,500 samples distributed across 3 classes, featuring a high-dimensional view (3,000D) and an extremely low-dimensional view (10D) with structurally masked complementary features.
- Real-World Benchmarks (Clustering/Classification): UCI Multiple Features, Caltech-USD Birds (CUB), ORL Face, MSRCV1, Mfeat, and 100Leaves.
- DEAP: An emotion recognition dataset presenting a natural extreme imbalance (Video: $\approx 1.5 \times 10^5$D vs. EEG: 32D and Peripheral signals: 8D).
- NYU-Depth V2 (NYUv2): Used to test the model's generalization on a downstream semantic segmentation task.
Key Insights and Ideas
- The Danger of Blind Expansion: The authors empirically demonstrate that expanding a 30-dimensional view into a 1024-dimensional space using standard dense DNNs causes validation accuracy to fluctuate violently while training error drops, a classic hallmark of overfitting.
- Compression vs. Expansion Trade-off: While compressing high-dimensional data down to low-dimensional spaces avoids overfitting, it causes massive information loss. AdaMuS proves that projecting to a high-dimensional space combined with structured pruning preserves the fine-grained details of large views while protecting small views.
- Granular Feature Selection via BN Scaling: Treating alignment noise as equivalent to genuine features erodes the uniqueness of smaller views. Using the scaling factor in Batch Normalization as a sparsity controller allows the network to dynamically deactivate individual dimensions that lack practical meaning.
Key Results
AdaMuS consistently outperformed 13 state-of-the-art baseline methods (including joint representation, graph-based, and decision-level fusion techniques) across all datasets.
- Clustering & Classification Accuracy: On the highly unbalanced CUB dataset, AdaMuS achieved a clustering accuracy of 81.65%, outperforming the nearest specialized imbalance baseline, UMDL (80.12%). On the naturally unbalanced DEAP dataset, it achieved 98.46% classification accuracy.
- Efficiency Gains: Compared to UMDL, AdaMuS dramatically reduced computational overhead. On the CUB dataset, it utilized only 39.39% of the parameters and 44.73% of the FLOPs of UMDL, proving that pruning redundant connections does not compromise representational capability.
- Downstream Transferability: When applied to semantic segmentation on NYUv2, AdaMuS-SEG achieved a mean Intersection over Union (mIoU) of 22.1, notably outperforming a standard standard UNet baseline (20.0 mIoU).
Limitations and Future Work
While AdaMuS provides a robust, parameter-free pruning strategy, its performance remains sensitive to the choice of downstream hyperparameters, specifically the sparsity controllers ($\lambda_1, \lambda_2$) and the number of neighbors ($K$) used in graph construction. The authors note that an overly strong sparsity constraint can lead to performance degradation, indicating that finding the optimal balance between feature selection and representation capacity still requires careful tuning during the self-supervised phase.
Summary updated