TL;DR
This paper introduces a catalog of "System User Interactive Test" (SUIT) smells, which are bad practices in GUI testing. They empirically analyze the prevalence and refactoring of these smells in industrial and open-source projects, finding that while some smells are common, refactoring is infrequent, suggesting a need for better tools and practices in GUI test maintenance.
What Problem Does This Paper Address?
Modern software applications heavily rely on Graphical User Interfaces (GUIs) for user interaction. Consequently, automated GUI testing, specifically System User Interactive Tests (SUITs), has become crucial for ensuring software quality. However, just like production code, test code can suffer from "test smells" – indicators of poor design and implementation choices that hinder maintainability. While test smells have been extensively studied for unit tests, research on SUITs, which interact with the application through its GUI, has been limited. This paper aims to bridge this gap by identifying, cataloging, and analyzing the prevalence and refactoring of test smells specifically within SUITs.
Methods
The authors employ a "multivocal literature review" to identify SUIT-specific smells, combining insights from both academic research and grey literature (e.g., white papers, blogs) to capture a broader perspective from practitioners. Based on this review, they propose a catalog of 35 SUIT smells. They then conduct an empirical analysis on a dataset of industrial and open-source test suites. For 16 of the identified smells, they develop automated detection metrics based on code analysis. These metrics are used to assess the prevalence of smell symptoms and the frequency of refactoring actions aimed at removing them.
Datasets
The study utilizes two sets of datasets:
- Industrial Project: 48 repositories from BGL BNP Paribas, a partner company.
- Open-Source Projects: 12 repositories mined from GitHub.
In total, the dataset comprises over 2.8 million SUITs.
Key Insights and Ideas
- Bridging the Academia-Practitioner Gap: The multivocal literature review is crucial for capturing a comprehensive understanding of SUIT smells, as practitioners often identify issues not yet formally documented in academic literature.
- SUITs Have Unique Smells: GUI interaction introduces specific challenges that lead to test smells distinct from those found in unit tests.
- Refactoring Lag: While many SUIT smells are prevalent, refactoring actions to address them are surprisingly infrequent, indicating a potential disconnect between identifying issues and actively fixing them.
- Smell Evolution: Some smells, like "Narcissistic," can disappear not through direct refactoring but as a side effect of replacing old, symptomatic tests with new ones.
Key Results
- The study identified a catalog of 35 SUIT-specific smells.
- Metrics were developed to automatically detect 16 of these smells.
- Hardcoded Values (over 90% of tests), Over Checking (75-80%), and Sneaky Checking (70%) were found to be the most prevalent SUIT smells across both industrial and open-source projects.
- Despite the prevalence of these smells, less than 50% of affected tests undergo refactoring.
- Missing Assertion is an exception, with a high refactoring rate (70-90%).
- Industrial projects showed a higher prevalence of smells related to complexity, such as Army of Clones and Middle Man, potentially due to their project structure and use of translation layers.
Limitations and Future Work
Limitations: * The study acknowledges a threat to construct validity due to the reliance on heuristics and code metrics for smell detection, which may not capture all instances of a smell. * The internal validity is potentially impacted by how code changes are extracted, as refactoring actions might be missed if not properly checked in. * External validity is limited by the specific projects analyzed, and the results may not generalize to all languages or technologies beyond Robot Framework.
Future Work: * Devise automated refactoring approaches for SUIT smells. * Extend the catalog of SUIT smells and the metrics used for detection. * Further investigate the impact of SUIT smells on development processes and explore how to improve awareness of these issues.
Summary updated