In clinical machine learning, we often assume every patient has the same set of data points. In reality, one patient might have a blood panel and a survey, while another has an MRI and clinical notes. This creates heterogeneous feature spaces, where the "columns" in our spreadsheet don't match up across groups.
A new paper from Duke University, Learning Across Heterogeneous Feature Spaces with Group Robustness, introduces a clever framework to handle this messiness. By rethinking what it means for a model to be "fair" and how different data types can "talk" to each other, the authors achieve state-of-the-art results with significantly fewer parameters.
TL;DR
The authors propose a framework that optimizes for worst-group excess loss—the gap between a group's current performance and its theoretical best—rather than raw loss. They enable knowledge transfer between groups with completely different features by mapping them into a shared latent space organized around learnable Gaussian class anchors.
What Problem Does This Paper Address?
Standard machine learning models struggle when data is missing or inconsistent across groups. If Group A has high-resolution imaging and Group B only has basic vitals, Group B will naturally have a higher "best possible" loss.
Existing "group robust" methods like Group Distributionally Robust Optimization (Group DRO) try to help the "worst" group by upweighting those with the highest raw loss. However, as shown in the figure below, this can be unfair: a high loss might just mean the features are uninformative, not that the model is poorly trained. Furthermore, if two groups share no features (disjoint spaces), most current methods fail to transfer any knowledge between them, essentially training separate, weaker models for each.
Methods
The paper introduces two main innovations:
- Minimax Excess Loss: Instead of minimizing the maximum raw loss, the framework minimizes the maximum excess loss. To do this, they first estimate the "optimal loss" ($L^*_g$) for each group using a nested cross-validation algorithm. This ensures the model focuses its "effort" on groups that can improve, rather than those limited by poor data quality.
- Shared Latent Space with Gaussian Anchors: To allow groups with different features to collaborate, the model uses group-specific encoders that map features into a common latent space $Z$.
- Alignment Loss: They learn a Gaussian "anchor" for each class (e.g., "Healthy" vs. "Sick"). An alignment loss pulls the embeddings of patients from every group toward the anchor of their respective class.
- Separation Loss: This ensures the anchors themselves stay distinct so the final classification head can easily tell them apart.
Datasets
The framework was evaluated on three diverse medical datasets:
- NHANES: Survey data from 17,005 adults to predict cardiovascular disease.
- Fed-Heart: Data from four different cardiology hospitals with varying clinical features.
- EMBED: A massive breast imaging dataset (128,680 records) where groups are defined by which of the four standard mammographic views are available.
The researchers also created "designed" versions of these datasets where feature spaces were forced to be fully disjoint to test the model's ability to transfer knowledge without shared "columns."
Key Insights and Ideas
- Fairness is Relative: The core "aha" moment is that raw loss is a misleading metric for robustness in heterogeneous settings. By using excess loss, the model acknowledges that some data sources are inherently noisier or less predictive than others.
- Collaboration via Structure: You don't need shared features to share knowledge. By forcing different encoders to map to the same class-based "neighborhoods" in a latent space, the model learns a universal representation of the disease state that transcends the specific tests performed.
- Efficiency Wins: Because the model uses a single shared head and structured latent space, it requires far fewer parameters than "Mixture of Experts" (MoE) approaches, which often scale linearly with the number of features or groups.
Key Results
The framework consistently outperformed state-of-the-art baselines like Flex-MoE and REMIND:
- Loss Reduction: On the EMBED dataset, it reduced worst-group loss by up to 79% compared to baselines.
- Excess Loss: It reduced worst-group excess loss by up to 95%.
- Parameter Efficiency: The model achieved these results with up to 18× fewer parameters than competing methods in disjoint feature settings.
- Robustness to Disjoint Features: Even when groups shared zero features, the Gaussian anchors allowed the model to maintain high accuracy by effectively "aligning" the different data types in the latent space.
Limitations and Future Work
The authors note that the success of the method relies on the reliability of the reference loss estimates. If a group is very small (e.g., the Switzerland hospital in Fed-Heart with only 123 patients), the estimated optimal loss can be noisy, which might lead to sub-optimal weighting during training.
Future work could explore more sophisticated ways to estimate these references or investigate how this structured latent space could be used for "transferring" a model to a entirely new group that wasn't seen during training.
Summary updated