Benchmarking Real-World Many-Objective Problems: A Problem Suite With Baseline Results

Vikas Palakonda, Jae-Mo Kang, Heechul JungIEEE Access 2024 · DOI · OpenReview

This paper, recently accepted for publication in IEEE Access, addresses a critical gap in the field of evolutionary computation: the disconnect between theoretical benchmarks and real-world engineering challenges.

TL;DR

The authors introduce a new benchmark suite of 11 complex, real-world "many-objective" optimization problems (MaOPs) sourced from diverse engineering fields. By testing eight state-of-the-art algorithms against these problems, they reveal that many popular methods designed for artificial benchmarks struggle when faced with the messy, constrained realities of actual engineering design.

What Problem Does This Paper Address?

In the world of optimization, there is a significant leap from "multi-objective" (2–3 goals) to "many-objective" (more than 3 goals). As the number of objectives increases, traditional algorithms often fail because almost every potential solution becomes "non-dominated"—meaning the algorithm can't easily decide which direction is "better."

Historically, researchers have tested their Many-Objective Evolutionary Algorithms (MOEAs) on artificial, mathematically "clean" test suites. However, these synthetic problems lack the rugged landscapes, complex constraints, and unpredictable trade-offs found in real engineering. Real-world problems are often locked behind proprietary doors or require expensive simulation software, making them hard for the broader research community to access. This paper democratizes access to 11 such problems, providing a standardized "Ma-RWP" (Many-Objective Real-World Problem) suite.

Methods

The researchers didn't just compile a list; they formalized the mathematical models for 11 distinct problems and established a rigorous benchmarking framework:

  • Reference Pareto Fronts: Since the "perfect" solutions for these real-world problems are unknown, the authors generated reference fronts by aggregating the best results from all tested algorithms and applying a MaxiMin sorting scheme to ensure a diverse, high-quality comparison set.
  • Evaluation Metrics: They used two gold-standard metrics: Hypervolume (HV), which measures the "size" of the objective space covered by the solutions, and Inverted Generational Distance (IGD), which measures how close and well-distributed the solutions are relative to the reference front.
  • Algorithm Selection: They benchmarked eight diverse algorithms, including Pareto-dominance-based (MultiGPO, Pi-MOEA), indicator-based (MaOEA-IBP), decomposition-based (MOEAD-URAW, AREA), and ensemble/hybrid methods (VMEF, TSNSGAII, PeEA).

Datasets

The core contribution is the Ma-RWP suite, consisting of 11 problems (Ma-RW1 through Ma-RW11) with objectives ranging from 4 to 10. These include:

  • Structural Engineering: Car side impact design and car cab design.
  • Aerospace/Marine: Liquid-rocket injector design and conceptual marine design.
  • Manufacturing: Machining processes and work roll cooling (single and multi-pass).
  • Environmental/Chemical: Water resource planning and pollution monitoring.
  • Electronics/Materials: Ultra-wideband antenna design and oil-repellent fabric development.

Key Insights and Ideas

  • The "Artificiality" Trap: Algorithms that excel on smooth, synthetic Pareto fronts often stumble on real-world problems. For instance, PeEA (which estimates the shape of the Pareto front) performed poorly, likely because real-world fronts are too irregular for simple estimation.
  • Indicator-Based Robustness: MaOEA-IBP emerged as a top performer. Because it uses an "indicator" to guide the search rather than relying on pre-defined reference points or geometric assumptions, it adapts better to the "weird" shapes of engineering trade-offs.
  • The Reference Point Struggle: Decomposition-based methods like AREA, which rely on reference points, showed poor performance. The authors suggest that finding suitable reference points for real-world applications is a major hurdle that synthetic benchmarks don't adequately simulate.
  • Selection Pressure is Key: In many-objective spaces, maintaining "selection pressure" (the drive toward the optimal front) is difficult. The algorithms that successfully modified their dominance criteria (like MultiGPO) or used weight-vector adjustments (like MOEAD-URAW) were significantly more effective.

Key Results

  • Hypervolume (HV) Winners: MultiGPO and MaOEA-IBP were the overall leaders, consistently finding solution sets that covered the largest portion of the objective space.
  • IGD Winners: MOEAD-URAW and Pi-MOEA proved best at finding solutions that were both close to the optimal front and well-distributed.
  • The Underperformers: PeEA and TSNSGAII consistently ranked at the bottom across almost the entire suite, highlighting a need to rethink "multi-stage" or "shape-estimation" approaches for non-synthetic data.

Limitations and Future Work

The authors acknowledge that while 11 problems are a great start, the suite needs to expand to include even more diverse disciplines to avoid "algorithmic overfitting" to specific types of engineering math.

Furthermore, because the "true" Pareto fronts are unknown, the baseline results are relative to the current state-of-the-art. As better algorithms emerge, the reference fronts will need to be updated. Future work will involve expanding the suite and exploring how these algorithms handle even higher-dimensional decision spaces (more variables) alongside the many objectives.

Summary updated

Abstract

In recent decades, multi-objective evolutionary algorithms (MOEAs) have been evaluated on artificial test problems with unrealistic characteristics, leading to uncertain conclusions about their efficacy in real-world applications. To address this issue, a few benchmark test suites comprising real-world problems have been proposed for MOEAs, encompassing numerous multi-objective problems and a select few many-objective problems. Given the distinct challenges posed by many-objective optimization problems (MaOPs) and their inherent difficulty, developing a test suite that includes real-world problems with many conflicting objectives is crucial. Hence, in this paper, we propose a comprehensive test suite for benchmarking the many-objective real-world complex problems. This test suite consists of 11 problems collected from different disciplines of engineering. Furthermore, we comprehensively analyzed the problems in our newly proposed test suite, employing eight state-of-the-art algorithms rooted in various fundamental principles specifically designed to address MaOPs. The experimental findings highlight the strong performance of indicator-based, weight-vector-based decomposition, Pareto-dominance-based, and hybrid MOEAs on the proposed test suite. In contrast, reference-vector-based decomposition approaches, Pareto front shape estimation-based methods, and multi-evolution approaches exhibit relatively weaker performance.
BibTeX
@article{palakonda2024benchmarking,
  author = {Vikas Palakonda and Jae-Mo Kang and Heechul Jung},
  title = {{Benchmarking Real-World Many-Objective Problems: A Problem Suite With Baseline Results}},
  journal = {IEEE Access},
  year = {2024},
  doi = {10.1109/access.2024.3383916},
}
Figure