How Practitioners Perceive the Relevance of ESEM Research

Jeffrey C. Carver, Oscar Dieste, Nicholas A. Kraft, David Lo, Thomas ZimmermannESEM 2016 · DOI

TL;DR

This paper investigates how industrial practitioners perceive the relevance of research presented at the Empirical Software Engineering and Measurement (ESEM) conference. The study found that ESEM research is generally viewed as relevant by practitioners, with a significant portion of ratings falling into "essential" or "worthwhile" categories. The research also highlights specific areas where practitioners desire more focus from the ESEM community.

What Problem Does This Paper Address?

A persistent challenge in software engineering is the "gap" between academic research and industry practice. While researchers often aim for their work to have a practical impact, it's crucial to understand if practitioners actually find this research relevant to their daily work. Low industrial participation in conferences can be a symptom of this disconnect. This paper specifically addresses this by asking: "Is research published at ESEM relevant to practitioners' needs, and how can ESEM authors make their research even more relevant?" This understanding is vital for the long-term health and impact of the ESEM conference and empirical software engineering research in general.

Methods

The authors replicated a previous study by Lo et al. [11] but expanded its scope. They surveyed a large number of industry practitioners, asking them to rate the relevance of research papers.

Here's a breakdown of their approach:

  • Paper Summarization: Instead of having practitioners read full papers, the researchers used short summaries. They employed a hybrid approach:
  • Author Summaries: Authors of ESEM papers were asked to provide concise, jargon-free summaries (under 240 characters) understandable to general practitioners.
  • Researcher Summaries: For papers where authors didn't provide summaries, the researchers created their own. They even created two types of researcher summaries: "Simple" and "Detailed" (including information about study type and participants) to investigate the impact of summary detail.
  • Survey Design: A web-based survey was used to collect feedback. Practitioners were presented with randomly selected paper summaries and asked to rate their relevance using categories: "Essential," "Worthwhile," "Unimportant," or "Unwise." They also gathered qualitative feedback on why certain ratings were given and what research topics practitioners would like to see addressed.
  • Data Analysis: Quantitative analysis involved calculating E-scores (percentage of "Essential" ratings), EW-scores (percentage of "Essential" or "Worthwhile" ratings), and U-scores (percentage of "Unwise" ratings). Qualitative data was analyzed using card sorting techniques to identify recurring themes.

Datasets

The study used a dataset of 161 Full Papers and Industrial Experience papers from the International Symposium on Empirical Software Engineering and Measurement (ESEM) conferences held between 2011 and 2015. For each paper, two summaries were created (either by authors or researchers), resulting in a total of 322 summaries. The survey received 9,941 ratings from 437 practitioners.

Key Insights and Ideas

  • ESEM Research is Generally Relevant: The most striking finding is that ESEM research is largely perceived as valuable by industrial practitioners. This counters potential assumptions that academic research is out of touch with industry needs.
  • Citation Count Doesn't Predict Practitioner Relevance: Surprisingly, the study found no correlation between a paper's citation count and how relevant practitioners found it. This suggests that traditional academic metrics for impact might not align with industry perception.
  • Summary Matters, But Not Dramatically: While author-written summaries tended to receive slightly higher ratings than researcher-written ones, the difference was not substantial. Similarly, simple summaries performed slightly better than detailed ones. This indicates that while the way research is presented is important, it doesn't drastically alter the perceived relevance.
  • Practitioners Want More Focus on Specific Areas: The qualitative feedback revealed clear desires for research to address specific problems in areas like Code Quality, Sustainability, Testing, Estimation, and understanding Customer needs.

Key Results

  • Overall Relevance: 67% of all ratings were "Essential" or "Worthwhile," while only 5% were "Unwise."
  • Demographic Differences: Practitioners with more experience and those in management roles tended to give slightly lower relevance ratings compared to others.
  • Industry Co-Authorship: Papers with industrial co-authors or those from the industry track showed only marginal, non-significant differences in ratings compared to purely academic papers.
  • Highly Rated Ideas: The top-rated research ideas often mapped directly to the areas practitioners identified as important, such as testing, developer productivity, and customer understanding. For instance, papers on "Cost effectiveness of unit testing" and "API usability" were highly rated.
  • Qualitative Guidance: The most frequently requested research areas by practitioners included Code Quality (15x), Sustainability (13x), Testing (13x), Estimation (12x), and understanding Customers (10x).

Limitations and Future Work

  • Summary Dependence: The study's findings are based on paper summaries, not the full papers. While efforts were made to ensure summaries were representative, this is an inherent limitation.
  • Company Bias: A significant portion of respondents came from Microsoft and ABB. While the study expanded beyond previous work, the perspectives of practitioners from other companies might differ.
  • Perceived Relevance vs. Adoption: The study measures perceived relevance, not actual adoption or impact. A highly relevant paper might still not be adopted due to other organizational factors.

Future work suggested includes:

  • Improving Discoverability: Finding ways to make ESEM research more discoverable to practitioners.
  • Streamlining Feedback: Exploring methods to integrate summary collection into the publication process and share practitioner feedback directly with authors.
  • Broader SE Community: Repeating this study across other software engineering communities to gain a wider understanding of practitioner perceptions.

Summary updated

BibTeX
@inproceedings{carver2016how,
  author = {Jeffrey C. Carver and Oscar Dieste and Nicholas A. Kraft and David Lo and Thomas Zimmermann},
  title = {{How Practitioners Perceive the Relevance of ESEM Research}},
  booktitle = {ESEM},
  year = {2016},
  doi = {10.1145/2961111.2962597},
}
Figure