Generator Assisted Mixture of Experts For Feature Acquisition in Batch

Vedang Asgaonkar, Aditya Jain, Abir De

Abstract

Given a set of observations, feature acquisition is about finding the subset of unobserved features which would enhance accuracy. Such problems have been explored in a sequential setting in prior work. Here, the model receives feedback from every new feature acquired and chooses to explore more features or to predict. However, sequential acquisition is not feasible in some settings where time is of the essence. We consider the problem of feature acquisition in batch, where the subset of features to be queried in batch is chosen based on the currently observed features, and then acquired as a batch, followed by prediction. We solve this problem using several technical innovations. First, we use a feature generator to draw a subset of the synthetic features for some examples, which reduces the cost of oracle queries. Second, to make the feature acquisition problem tractable for the large heterogeneous observed features, we partition the data into buckets, by borrowing tools from locality sensitive hashing and then train a mixture of experts model. Third, we design a tractable lower bound of the original objective. We use a greedy algorithm combined with model training to solve the underlying problem. Experiments with four datasets show that our approach outperforms these methods in terms of trade-off between accuracy and feature acquisition cost.

1. Introduction

In many real-world machine learning applications, data is often incomplete at the outset. For instance, in medical diagnostics, a clinician observes initial symptoms and must decide which additional tests to order to reach an accurate diagnosis. While prior research has focused on sequential acquisition—where features are selected one by one based on immediate feedback—this approach is often impractical due to time constraints. In scenarios where test results take days to process, a batch acquisition strategy is required, where a subset of features is selected and acquired simultaneously.

Batch feature acquisition presents significant challenges: the optimal subset depends on the initial observations, and the combinatorial search space is NP-hard. Furthermore, querying an "oracle" (e.g., a medical lab) is often prohibitively expensive. This paper introduces GENEX (Generator Assisted Mixture of Experts), a framework designed to optimize the trade-off between prediction accuracy and acquisition cost in batch settings.

2. The GENEX Framework

GENEX incorporates three primary technical innovations to address the complexities of batch feature acquisition:

  1. Feature Generation as a Surrogate: To minimize expensive oracle queries, GENEX employs a $\beta$-VAE (Variational Autoencoder) to synthesize missing feature values. If the model's confidence in these generated values exceeds a threshold, they are used for prediction, effectively bypassing the need for real data.
  2. Random Hyperplane Clustering: The heterogeneity of observed features makes standard clustering methods like K-means ineffective, often leading to imbalanced data partitions. GENEX utilizes Locality Sensitive Hashing (LSH) via random hyperplanes to group similar instances into "buckets" more equitably.
  3. Mixture of Experts (MoE): Specialized "expert" models are trained for each bucket, allowing the system to master specific regions of the feature space.
Comparison of clustering methods (RH + ANN vs. K-means and Gaussian Mixture) across DP and CIFAR100 datasets, illustrating the superior performance of Random Hyperplane clustering.

To optimize feature selection, the framework utilizes a greedy algorithm supported by a theoretical foundation extending the concepts of partial monotonicity and weak submodularity, ensuring near-optimal performance under specific conditions.

3. Experimental Evaluation

The performance of GENEX was evaluated across four diverse datasets: Disease Prediction (DP), MNIST, CIFAR100, and TinyImageNet.

3.1 Key Results

GENEX consistently outperformed six state-of-the-art baselines, including JAFA, EDDI, and ACFlow. A critical finding was the framework's cost efficiency; GENEX achieved accuracy levels comparable to oracle-only methods while reducing the number of real queries by 3–5x. This efficiency is largely attributed to the feature generator's ability to provide "good enough" synthetic data for confident predictions.

Extracted figure from page 7 of Generator Assisted Mixture of Experts For Feature Acquisition in Batch

3.2 Scalability and Robustness

Unlike reinforcement learning-based approaches that often struggle with high-dimensional data, GENEX demonstrated strong scalability, particularly on the TinyImageNet dataset. The use of Random Hyperplane (RH) clustering was also shown to be superior to K-means, as it encourages cluster diversity and ensures a well-distributed mixture of experts.

4. Conclusion and Future Work

GENEX provides a robust solution for batch feature acquisition by leveraging synthetic data and specialized experts. While the framework offers significant improvements in accuracy and cost, future work could address current limitations, such as the assumption of uniform feature costs. Incorporating instance-specific costs into the optimization objective remains a promising direction for further research.

Summary updated

Abstract

Given a set of observations, feature acquisition is about finding the subset of unobserved features which would enhance accuracy. Such problems have been explored in a sequential setting in prior work. Here, the model receives feedback from every new feature acquired and chooses to explore more features or to predict. However, sequential acquisition is not feasible in some settings where time is of the essence. We consider the problem of feature acquisition in batch, where the subset of features to be queried in batch is chosen based on the currently observed features, and then acquired as a batch, followed by prediction. We solve this problem using several technical innovations. First, we use a feature generator to draw a subset of the synthetic features for some examples, which reduces the cost of oracle queries. Second, to make the feature acquisition problem tractable for the large heterogeneous observed features, we partition the data into buckets, by borrowing tools from locality sensitive hashing and then train a mixture of experts model. Third, we design a tractable lower bound of the original objective. We use a greedy algorithm combined with model training to solve the underlying problem. Experiments with four datasets show that our approach outperforms these methods in terms of trade-off between accuracy and feature acquisition cost.
BibTeX
@inproceedings{asgaonkar2024generator,
  author = {Vedang Asgaonkar and Aditya Jain and Abir De},
  title = {{Generator Assisted Mixture of Experts For Feature Acquisition in Batch}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=miSAnyBlbC},
}
Figure