Joint Optimization of Features and Hyperparameters

27 concepts18 papers1 proposal

Traditional machine learning workflows treat feature selection and hyperparameter tuning as an assembly line. First, you filter out irrelevant or redundant features using statistical tests or a proxy model. Then, you pass this pruned dataset to your primary algorithm to tune its hyperparameters, such as regularization strength or tree depth.

As de Lacy et al. argue, this sequential separation is fundamentally flawed. It assumes that the optimal feature set is independent of the model's internal settings. In reality, data representation and model capacity are deeply coupled. Isolating them leads to suboptimal models because a feature set that excels under one hyperparameter configuration might perform poorly under another.

The Coupling Problem

Why do features and hyperparameters depend on each other? Consider a Support Vector Machine (SVM). Its performance is highly sensitive to the interaction between the input features and its hyperparameters, such as the kernel coefficient ($\gamma$) or the regularization parameter ($C$) (Lou et al.).

If you select features first using a default hyperparameter configuration, you might discard a feature that would be highly predictive if $\gamma$ were scaled down. Conversely, tuning hyperparameters on an unpruned feature space forces the model to adapt its parameters to suppress noise rather than maximize signal. For example, a model might increase its regularization penalty to combat noisy features, which inadvertently suppresses the signal from useful ones. As Haddow notes, decoupling the search for optimal features from classifier settings limits generalization. Solving these two problems separately fails to capture the balance between model complexity and predictive accuracy (Lou et al.).

The Unified Search Space

To resolve this, joint optimization collapses feature selection and hyperparameter tuning into a single loop. Instead of searching two separate spaces sequentially, the algorithm navigates a unified space.

If a dataset has $d$ candidate features and the model has $k$ continuous hyperparameters, the joint search space becomes a mixed-integer domain:

$$\mathcal{X} = \{0, 1\}^d \times \mathbb{R}^k$$

A single point in this space defines both what the model sees (a binary mask of feature inclusions) and how the model behaves (the continuous hyperparameters).

Sequential Pipeline:
[ All Features ] ──> ( Feature Selection ) ──> [ Reduced Features ] ──> ( Hyperparameter Tuning ) ──> Final Model

Joint Optimization Loop:
                      ┌──────────────────────────────────────┐
                      ▼                                      │
[ Unified Vector: [ Binary Mask ── Continuous Params ] ] ──> ( Evaluate Fitness )

By evaluating points in this combined space, the optimization framework can discover non-linear synergies between data inputs and model architecture.

Algorithmic Strategies for Co-Optimization

Navigating this hybrid, high-dimensional space is challenging. Researchers use two primary paradigms to search it efficiently: metaheuristics and Bayesian optimization.

Evolutionary and Swarm Algorithms

Metaheuristic algorithms are well-suited for joint optimization because their representations can easily concatenate discrete and continuous variables.

  • Genetic Algorithms (GAs): In a GA framework, a single chromosome represents both the binary feature mask and the continuous hyperparameters. Dutartre et al. used this approach to evolve a combined genotype for reservoir computing, showing that mutating the joint vector preserves feature diversity and prevents overfitting. Similarly, Zhao et al. and De Clercq et al. leveraged hybrid GA wrappers to evaluate feature-parameter pairs based on classification accuracy.
  • Swarm Intelligence: Other approaches use particle swarms or bio-inspired algorithms to explore the space. El-Attar et al. combined Particle Swarm Optimization (PSO) for feature selection with GAs for hyperparameter tuning to navigate anomaly detection spaces. Wang et al. applied a single metaheuristic algorithm to handle both tasks simultaneously, ensuring the classifier's decision boundaries perfectly align with the reduced feature space.

Bayesian Optimization

An alternative approach is to treat feature selection as a hyperparameter within a Bayesian Optimization (BO) framework. Rautenstrauß et al. demonstrated this by treating feature inclusion/exclusion as a categorical hyperparameter. This creates a high-dimensional search space that advanced BO techniques explore using a probabilistic surrogate model, balancing exploration and exploitation across both structural settings and data inputs simultaneously.

Why It Works: Co-Adaptation and Synergy

The primary advantage of joint optimization is co-adaptation. When the feature selection mechanism and the hyperparameter tuner communicate in a single loop, they tailor the model architecture to the exact data representation it receives (Wang et al.).

This integration creates a clear synergy (Javeed et al.). For example, in medical diagnostics, refining both the input data and the model's internal settings simultaneously yields significantly higher predictive metrics than isolating the two steps (Cao et al.). It prevents the fragmented tuning that often occurs in complex systems trying to balance local accuracy and global consistency (Stymne et al.), and helps bypass the limitations of basic feature selection strategies (Lefever et al.).

Practical Challenges

While joint optimization offers clear theoretical and empirical benefits, it introduces two main challenges:

  1. Search Space Explosion: Combining features and hyperparameters expands the search space exponentially. Finding the global optimum requires significant computational overhead, often requiring parallelization or highly optimized evaluation steps.
  2. Boundary Sensitivity: The success of the optimization depends heavily on the initial ranges provided by the practitioner. As Saiapin et al. note, if the optimal hyperparameter values fall outside the predefined candidate set, the model's ultimate predictive performance remains constrained, regardless of how well the feature selection mechanism performs.

Key Foundational Work

A novel approach to extracting features from motif content and protein composition for protein sequence classification (2005) — contributed 2 concept elements to this direction.
Using Automated Feature Optimisation to Create an Adaptable Relation Extraction System (2008) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_joint_feature_and_hyperparameter_optimiz_13ee0e45,
  title = {joint feature and hyperparameter optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-feature-and-hyperparameter-optimization/rdr_13ee0e45f85d49f5993ba12d74f5f157},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_joint_feature_and_hyperparameter_optimiz_13ee0e45_v1,
  title = {joint feature and hyperparameter optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-feature-and-hyperparameter-optimization/rdr_13ee0e45f85d49f5993ba12d74f5f157/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake