Smells in System User Interactive Tests

Renaud Rwemalika, Sarra Habchi, Mike Papadakis, Yves Le Traon, Marie-Claude Brasseur

TL;DR

This paper introduces a catalog of "System User Interactive Test" (SUIT) smells, which are bad practices in GUI testing. They empirically analyze the prevalence and refactoring of these smells in industrial and open-source projects, finding that while some smells are common, refactoring is infrequent, suggesting a need for better tools and practices in GUI test maintenance.

What Problem Does This Paper Address?

Modern software applications heavily rely on Graphical User Interfaces (GUIs) for user interaction. Consequently, automated GUI testing, specifically System User Interactive Tests (SUITs), has become crucial for ensuring software quality. However, just like production code, test code can suffer from "test smells" – indicators of poor design and implementation choices that hinder maintainability. While test smells have been extensively studied for unit tests, research on SUITs, which interact with the application through its GUI, has been limited. This paper aims to bridge this gap by identifying, cataloging, and analyzing the prevalence and refactoring of test smells specifically within SUITs.

Methods

The authors employ a "multivocal literature review" to identify SUIT-specific smells, combining insights from both academic research and grey literature (e.g., white papers, blogs) to capture a broader perspective from practitioners. Based on this review, they propose a catalog of 35 SUIT smells. They then conduct an empirical analysis on a dataset of industrial and open-source test suites. For 16 of the identified smells, they develop automated detection metrics based on code analysis. These metrics are used to assess the prevalence of smell symptoms and the frequency of refactoring actions aimed at removing them.

Datasets

The study utilizes two sets of datasets:

  • Industrial Project: 48 repositories from BGL BNP Paribas, a partner company.
  • Open-Source Projects: 12 repositories mined from GitHub.

In total, the dataset comprises over 2.8 million SUITs.

Key Insights and Ideas

  • Bridging the Academia-Practitioner Gap: The multivocal literature review is crucial for capturing a comprehensive understanding of SUIT smells, as practitioners often identify issues not yet formally documented in academic literature.
  • SUITs Have Unique Smells: GUI interaction introduces specific challenges that lead to test smells distinct from those found in unit tests.
  • Refactoring Lag: While many SUIT smells are prevalent, refactoring actions to address them are surprisingly infrequent, indicating a potential disconnect between identifying issues and actively fixing them.
  • Smell Evolution: Some smells, like "Narcissistic," can disappear not through direct refactoring but as a side effect of replacing old, symptomatic tests with new ones.

Key Results

  • The study identified a catalog of 35 SUIT-specific smells.
  • Metrics were developed to automatically detect 16 of these smells.
  • Hardcoded Values (over 90% of tests), Over Checking (75-80%), and Sneaky Checking (70%) were found to be the most prevalent SUIT smells across both industrial and open-source projects.
  • Despite the prevalence of these smells, less than 50% of affected tests undergo refactoring.
  • Missing Assertion is an exception, with a high refactoring rate (70-90%).
  • Industrial projects showed a higher prevalence of smells related to complexity, such as Army of Clones and Middle Man, potentially due to their project structure and use of translation layers.

Limitations and Future Work

Limitations: * The study acknowledges a threat to construct validity due to the reliance on heuristics and code metrics for smell detection, which may not capture all instances of a smell. * The internal validity is potentially impacted by how code changes are extracted, as refactoring actions might be missed if not properly checked in. * External validity is limited by the specific projects analyzed, and the results may not generalize to all languages or technologies beyond Robot Framework.

Future Work: * Devise automated refactoring approaches for SUIT smells. * Extend the catalog of SUIT smells and the metrics used for detection. * Further investigate the impact of SUIT smells on development processes and explore how to improve awareness of these issues.

Summary updated

Abstract

Test smells are known as bad development practices that reflect poor design and implementation choices in software tests. Over the last decade, there are few attempts to study test smells in the context of system tests that interact with the System Under Test through a Graphical User Interface. To fill the gap, we conduct an exploratory analysis of test smells occurring in System User Interactive Tests (SUIT). We thus, compose a catalog of 35 SUIT-specific smells, identified through a multi-vocal literature review, and show how they differ from smells encountered in unit tests. We also conduct an empirical analysis to assess the diffuseness and removal of these smells in 48 industrial repositories and 12 open-source projects. Our results show that the same type of smells tends to appear in both industrial and open-source projects, but they are not addressed in the same way. We also find that smells originating from a combination of multiple code locations appear more often than those that are localized on a single line. This happens because of the difficulty to observe non-local smells without tool support. Furthermore, we find that smell-removing actions are not frequent with less than 50% of the affected tests ever undergoing a smell removal. Interestingly, while smell-removing actions are rare, some smells disappear while discarding tests, i.e., these smells do not appear in follow-up tests that replace the discarded ones.
BibTeX
@inproceedings{rwemalika2021smells,
  author = {Renaud Rwemalika and Sarra Habchi and Mike Papadakis and Yves Le Traon and Marie-Claude Brasseur},
  title = {{Smells in System User Interactive Tests}},
  booktitle = {Empir. Softw. Eng},
  year = {2021},
  doi = {10.1007/s10664-022-10251-1},
}
Figure