Finding High-Dimensional D-Optimal Designs for Logistic Models via Differential Evolution

Weinan Xu, Weng Kee Wong, Kay Chen Tan, Jianxin XuIEEE Access 2019 · DOI

Abstract

D-optimal designs are frequently used in controlled experiments to obtain the most accurate estimate of model parameters at minimal cost. Finding them can be a challenging task, especially when there are many factors in a nonlinear model. As the number of factors becomes large and interact with one another, there are many more variables to optimize and the D-optimal design problem becomes high-dimensional and non-separable. Consequently, premature convergence issues arise. Candidate solutions get trapped in local optima and the classical gradient-based optimization approaches to search for the D-optimal designs rarely succeed. We propose a specially designed version of differential evolution (DE) which is a representative gradient-free optimization approach to solve such high-dimensional optimization problems. The proposed specially designed DE uses a new novelty-based mutation strategy to explore the various regions in the search space. The exploration of the regions will be carried out differently from the previously explored regions and the diversity of the population can be preserved. The proposed novelty-based mutation strategy is collaborated with two common DE mutation strategies to balance exploration and exploitation at the early or medium stage of the evolution. Additionally, we adapt the control parameters of DE as the evolution proceeds. Using logistic models with several factors on various design spaces as examples, our simulation results show our algorithm can find D-optimal designs efficiently and the algorithm outperforms its competitors. As an application, we apply our algorithm and re-design a 10-factor car refueling experiment with discrete and continuous factors and selected pairwise interactions. Our proposed algorithm was able to consistently outperform the other algorithms and find a more efficient D-optimal design for the problem.

Introduction: The Challenge of D-Optimal Design

In experimental science, researchers aim to obtain the most accurate parameter estimates for the lowest possible cost. D-optimal designs are a powerful tool for achieving this, as they minimize the volume of the confidence ellipsoid for model parameters. However, finding these designs for nonlinear models, such as logistic regression, is notoriously difficult—especially as the number of experimental factors grows.

High-dimensional design problems are characterized by three main difficulties:

  1. High Dimensionality: There are often hundreds of variables (support points and weights) to optimize simultaneously.
  2. Non-Separability: Interactions between factors mean variables cannot be optimized in isolation.
  3. Local Optima: The optimization landscape is "rugged," causing standard gradient-based or metaheuristic algorithms to suffer from premature convergence.

The NovDE Framework: Novelty and Adaptation

To overcome these hurdles, this research introduces NovDE (Novelty-based Differential Evolution). This specialized algorithm enhances the standard DE framework with several key innovations designed to maintain population diversity and navigate complex search spaces.

Novelty-Based Mutation

The core innovation of NovDE is its novelty-based mutation strategy. Unlike standard mutation operators that move toward known "good" areas (exploitation), this strategy explicitly rewards exploration. As shown in the figure below, the algorithm samples multiple potential difference vectors and selects the one ($d^*_{i,g}$) that has the largest angular difference ($\theta^s$) from the direction taken in the previous generation ($d_{i,g-1}$).

The novelty-based mutation strategy in NovDE selects the difference vector with the largest angular difference from the previous generation to promote exploration.

Hybrid Strategy and Structural Adaptation

NovDE balances exploration and exploitation by splitting the population. While one portion uses the novelty strategy, others utilize standard DE strategies like 'DE/rand/2'. Furthermore, the algorithm employs Multiple Exponential Recombination (MER). Unlike standard binomial crossover, MER updates consecutive variables together. This is crucial for experimental design, where support points are groups of related coordinates; updating them as blocks preserves the internal logic of candidate designs.

Finally, the crossover rate ($CR$) is not fixed. It adapts based on a "memory" of successful values from previous generations, allowing the algorithm to self-tune as the search progresses.

Experimental Validation

Synthetic Benchmarks

The algorithm was evaluated against logistic regression models with 7 continuous factors across various design spaces (e.g., $[-1, 1]^7$, $[-3, 3]^7$). In these tests, NovDE achieved success rates (finding designs with >99.99% efficiency) of 80-100% in most scenarios, significantly outperforming competitors like JADE or DDE-AMS, which often fell below 50% success.

Real-World Application: Car Refueling Experiment

The researchers applied NovDE to a complex 10-factor car refueling experiment involving a mix of discrete factors (e.g., "Ring Type"), continuous factors (e.g., "Lighting Angle"), and pairwise interactions. NovDE found a design with 95% D-efficiency, outperforming six other state-of-the-art DE variants and providing a more cost-effective experimental setup than previously possible.

Conclusion and Future Outlook

NovDE demonstrates that explicitly maintaining diversity through novelty-based movement is a powerful defense against premature convergence in high-dimensional design. While the current study focuses on logistic models, the success of the algorithm suggests it could be extended to other link functions (like Probit or Poisson) and multi-objective design problems. Future work will also explore the algorithm's performance in models with higher-order interactions, which represent the next frontier in experimental design complexity.

Summary updated

BibTeX
@article{xu2019finding,
  author = {Weinan Xu and Weng Kee Wong and Kay Chen Tan and Jianxin Xu},
  title = {{Finding High-Dimensional D-Optimal Designs for Logistic Models via Differential Evolution}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2018.2890593},
}
Figure