Exploring Performance Trade-offs in JHipster

Édouard Guégain, Alexandre Bonvoisin, Clément Quinton, Acher, Mathieu, Romain RouvoyarXiv 2024 · DOI · arXiv

TL;DR

This paper dives deep into the performance implications of configuration choices in JHipster, a popular web application generator. By exhaustively analyzing over 100,000 potential configurations, the authors reveal significant trade-offs between performance metrics like response time, size, and power consumption, demonstrating that optimizing for one often impacts others. They provide valuable insights for developers to make informed decisions and suggest avenues for automated performance optimization.

What Problem Does This Paper Address?

Modern software development faces a growing challenge: balancing diverse performance requirements. While traditional metrics like response time and binary size have always been crucial, power consumption has emerged as a significant concern, especially in cloud environments. This complexity is amplified by configurable software systems, which can generate a vast number of variations, making it impractical to test them all. Developers often resort to default settings or limited subsets of configurations, potentially missing out on optimal performance. This paper tackles this by investigating how specific configuration choices within a widely-used, highly configurable system (JHipster) impact various performance indicators, aiming to provide a systematic understanding of these trade-offs.

Methods

The researchers employed an empirical, experience-paper approach. They first constructed a detailed feature model of JHipster, identifying 118 valid configurations by excluding options not directly related to production performance (e.g., build tools, development databases). For each of these configurations, they meticulously measured a suite of performance indicators, including:

  • Static Metrics: Binary size, number of services, number of selected features, and boot time.
  • Response Times: Measured using Gatling for various operations like authentication, data retrieval (getall, get), creation (create), and deletion (delete).
  • Power Consumption: Assessed using RAPL for idle and workload states, broken down into CPU and RAM usage, and total power.

They then analyzed the collected data to identify correlations between indicators, understand the impact of individual options, and identify configurations that optimize specific performance goals.

Datasets

The study did not use pre-existing datasets for training. Instead, the researchers generated their own dataset by systematically configuring and testing 118 JHipster applications. This dataset, along with the entire replication package, is publicly available, allowing for reproducibility and further research.

Key Insights and Ideas

  • Configuration is King (for Performance): The study unequivocally demonstrates that JHipster configurations can lead to vastly different performance outcomes. Factors like database choice and search engine integration can drastically alter response times, size, and power consumption.
  • Interconnected Performance Metrics: Performance indicators are not isolated. Strong correlations exist, particularly within groups (e.g., CPU and RAM power usage are highly correlated). This suggests that optimizing one metric might indirectly benefit or harm others.
  • Proxy Indicators are Useful, But Limited: Certain indicators can serve as proxies for others (e.g., size for boot-time). This can simplify performance assessment by reducing the number of measurements needed. However, these proxies are not perfect and don't eliminate the need for specific evaluations.
  • Individual Options Have Uneven Impacts: A single configuration option can have a profound effect on one performance metric while barely influencing another. For example, the MSSQL database significantly increases boot time but has a less dramatic impact on response times compared to other databases.
  • Trade-offs are Inevitable: Optimizing for a single performance goal (e.g., lowest power consumption) often leads to compromises in other areas (e.g., slightly increased response times or size). This highlights the need for multi-objective optimization.

Key Results

  • Significant Performance Variation: The study found that workload power usage varied by a factor of 3, size by a factor of 4, and boot times could be up to 11 times longer across different configurations. Total idle power usage showed even greater variation, with factors as high as 51.
  • Strong Correlations Identified:
    • Size is strongly correlated with boot-time (0.72).
  • Create and delete response times are highly correlated (0.9).
  • CPU and RAM power usage are strongly correlated (0.95 idle, 0.96 under load).
  • Size is strongly correlated with all energetic indicators (0.76-0.85).
  • Impact of Specific Options:
    • MSSQL database significantly increases boot time (33.76s vs. ~5s for others) and size.
  • CouchbaseSE exhibits unusual power behavior: higher idle power than under load due to opportunistic indexing.
  • Elasticsearch increases size by 65% and boot time by 8%, and substantially increases power usage (41% idle, 20% load).
  • Reactive option reduces idle power by 6% but increases response times significantly (e.g., getall by 102%).
  • Near-Optimal Configurations: The paper identified configurations that were near-optimal across multiple indicators, such as C_power and C_size, suggesting these are good candidates for balancing performance goals.

Limitations and Future Work

  • Generalizability: The study acknowledges that its results are based on a specific, synthetic workload and a single data model. The performance of options might differ with actual production environments and diverse workloads.
  • Limited Scope: Certain JHipster options (e.g., Oracle database, micro-services, deployment options) were excluded due to licensing or benchmarking tool limitations, meaning their performance impact is not assessed.
  • Future Work: The authors suggest that more formal methods and accurate performance models are needed to fully automate configuration selection. They also highlight the potential for multi-objective optimization techniques to find configurations that effectively balance multiple performance goals, rather than optimizing for a single indicator in isolation.

Summary updated

Abstract

The performance of software systems remains a persistent concern in the field of software engineering. While traditional metrics like binary size and execution time have long been focal points for developers, the power consumption concern has gained significant attention, adding a layer of complexity to performance evaluation. Configurable software systems, with their potential for numerous configurations, further complicate this evaluation process. In this experience paper, we examine the impact of configurations on performance, specifically focusing on the web stack generator JHipster. Our goal is to understand how configuration choices within JHipster influence the performance of the generated system. We undertake an exhaustive analysis of JHipster by examining its configurations and their effects on system performance. Additionally, we explore individual configuration options to gauge their specific influence on performance. Through this process, we develop a comprehensive performance model for JHipster, enabling us to automate the identification of configurations that optimize specific performance metrics. In particular, we identify configurations that demonstrate near-optimal performance across multiple indicators and report on significant correlations between configuration choices within JHipster and the performance of generated systems.
BibTeX
@misc{guegain2024exploring,
  author = {Édouard Guégain and Alexandre Bonvoisin and Clément Quinton and Acher, Mathieu and Romain Rouvoy},
  title = {{Exploring Performance Trade-offs in JHipster}},
  year = {2024},
  eprint = {2409.16480},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.16480},
}
Figure