TL;DR
This paper introduces a lightweight survey framework to gauge how software engineering practitioners perceive the relevance of academic research. The study, conducted at Microsoft, found that practitioners generally view research positively, but also identified specific reasons why some research ideas are considered "unwise" and highlighted a lack of correlation between academic citations and practitioner relevance.
What Problem Does This Paper Address?
The field of software engineering research has seen a significant increase in publications. This raises a crucial question: how relevant is this research to the practitioners who are on the front lines of software development? While academic impact is often measured by citations, this doesn't necessarily reflect practical utility. The authors aim to bridge this gap by directly soliciting feedback from industry professionals to understand their perception of research relevance and identify areas for improvement in both research and its dissemination.
Methods
The core of this paper is a novel survey-based framework designed for rapid feedback. The process involves:
- Paper Selection and Summarization: Researchers selected 571 papers from top software engineering conferences (ICSE, ESEC/FSE, FSE) over a five-year period. For each paper, they created short, concise summaries (shorter than abstracts) that capture the key research ideas. This step was crucial to minimize the time investment for participants.
- Participant Selection: 3,000 full-time Microsoft employees in technical roles (developers, testers, program managers) were invited to participate. The selection aimed for a diverse representation of roles.
- Feedback Elicitation: An online survey was used. Participants were presented with randomly selected paper summaries and asked to rate their relevance on a Likert-like scale: "Essential," "Worthwhile," "Unimportant," "Unwise," or "I Don't Understand." A follow-up question allowed participants to explain why they rated an idea as "Unwise."
- Data Analysis: Quantitative metrics like E-score (percentage of "Essential" ratings) and EW-score (percentage of "Essential" or "Worthwhile" ratings) were computed. Qualitative analysis of free-text responses was used to understand the reasons behind "Unwise" ratings.
Datasets
The study utilized a dataset of 571 research papers published in the ICSE (2010-2014), ESEC/FSE (2009, 2011, 2013), and FSE (2010, 2012) conferences. The data collected was from 512 practitioners at Microsoft, who provided 17,913 ratings.
Key Insights and Ideas
- Practitioners are generally positive: A significant majority (71%) of ratings were "Essential" or "Worthwhile," indicating that practitioners do value the work produced by the software engineering research community.
- Experience matters: As practitioners' experience increased, they tended to be more critical, rating fewer studies as "essential." This suggests that seasoned professionals may have a more refined understanding of what is truly impactful.
- Role-based differences: Testers were found to be more positive about research relevance than developers or program managers. Program managers were also more critical than individual contributors.
- No correlation with citations: Surprisingly, the study found no statistically significant correlation between the number of citations a paper received and its perceived relevance by practitioners. This suggests that citation counts may not be a reliable indicator of practical impact.
- "Unwise" ideas reveal pitfalls: The qualitative analysis of "Unwise" ratings provided valuable insights into common reservations, such as tools being perceived as unnecessary, studies lacking actionable insights, generalizability issues, scalability concerns, and cost-benefit imbalances.
Key Results
- Overall EW-score: 71% of all ratings were "Essential" or "Worthwhile."
- Overall E-score: 20% of all ratings were "Essential."
- Demographic variations: Testers had an EW-score of 76%, while those with high experience had an EW-score of 64%.
- "Unwise" ratings: 3.4% of all votes were "Unwise."
- "I Don't Understand" ratings: 15.3% of ratings were "I Don't Understand," highlighting the need for clear summaries and the diverse technical backgrounds of practitioners.
Limitations and Future Work
The authors acknowledge several limitations:
- Summary quality: The findings depend on the quality of the paper summaries created by the researchers. While efforts were made to ensure clarity, some participants might have misunderstood certain summaries.
- Single company perspective: The study was conducted solely at Microsoft. Perceptions of research relevance might differ in other companies or industries.
- Relevance vs. Adoption: Perceived relevance does not equate to adoption. Adoption is influenced by many external factors (social, cultural, etc.) that were not assessed.
Future work suggested includes:
- Replicating the survey: Conducting similar surveys in other companies and across different countries to gain a broader perspective.
- Expanding to open-source developers: Investigating the relevance perceptions of open-source developers, who may have different motivations.
- Establishing practitioner panels: Creating representative panels of industry practitioners from diverse backgrounds for ongoing feedback.
- Integrating feedback into conferences: Partnering with conference organizers to regularly collect and utilize practitioner feedback to improve conference content and research direction.
Summary updated