Analyzing a practitioner perspective on relevance of published empirical research in Requirements Engineering

Niek TaxarXiv 2014 · arXiv

Abstract

The gap between academic research and industrial practice in Information Systems (IS) remains a significant hurdle. This paper evaluates a checklist designed to measure the practical relevance of research papers, specifically within the domain of Requirements Engineering (RE). Through an empirical study involving five practitioners and 24 research papers, we assess the inter-rater reliability of the checklist and the correlation between academic impact and practical utility. Our findings reveal a "poor" level of agreement among raters due to significant linguistic ambiguity and subjective interpretations. Additionally, we find no statistically significant correlation between citation counts and practitioner relevance scores, suggesting that scientific rigor does not necessarily translate to industry value.

Introduction

In Information Systems, "scientific rigor" often takes precedence over "practical relevance." While various instruments have been proposed to evaluate the utility of research for practitioners, these tools are rarely validated for usability. This study, conducted by Niek Tax, investigates the reliability of a checklist proposed by Kitchenham et al. and revised by Daneva et al. The goal is to determine if practitioners can reach a consensus when using such a tool to rate the relevance of research in Requirements Engineering.

Methods

The study employed five Master’s level students specializing in Computer Science or Business Information Technology to act as "practitioners." These participants possessed significant academic knowledge and some practical exposure to Requirements Engineering. The methodology consisted of three main pillars:

  1. Evaluation: Each participant applied a 20-question checklist to a set of 24 empirical research papers focused on Software Requirement Specification (SRS) comprehensibility.
  2. Statistical Analysis: Fleiss’ kappa was calculated to determine the degree of agreement among multiple raters. Additionally, Kendall’s tau measured the correlation between the papers' relevance scores and their scientific impact, determined by citation counts via Scopus.
  3. Qualitative Reconstruction: By analyzing the comments left by the raters, the author attempted to "reverse-engineer" why different people interpreted the same questions in wildly different ways, identifying specific linguistic and conceptual hurdles.

Results and Discussion

The results highlight a significant "agreement crisis." For 15 out of 19 questions, the agreement was negative, indicating less consensus than random chance. Only two questions reached "slight agreement": Q2 ("Is it claimed that the results are industry-relevant?") and Q5 ("Is it explicitly stated what the implications... for practice are?").

Inter-rater agreement (Fleiss’ kappa) for the two questions with the highest consensus.

The qualitative data revealed that terms like "believable evidence" were interpreted inconsistently—some raters sought statistical significance, while others relied on intuition. This subjectivity renders the resulting "relevance score" largely meaningless. Furthermore, the checklist's inconsistent use of active and passive voice likely hindered comprehensibility. Drawing on writing style guides, the author suggests this linguistic friction contributed to the poor performance of the tool.

Crucially, the study found no link between academic success and practical utility. The Kendall’s tau coefficient was -0.0988 (p=0.526), indicating that highly cited papers are not necessarily perceived as more relevant by practitioners.

Statistical correlation between citation counts and practitioner relevance scores.

Conclusion

The current checklist for assessing research relevance is hampered by subjectivity and linguistic friction. The findings suggest that the current tool is not fit for purpose in its current form. To bridge the rigor-relevance gap, evaluation tools must move toward objective, nominal categories. Future efforts should focus on reformulating these instruments to ensure consistent results across different evaluators, ultimately helping researchers produce work that is both scientifically sound and industrially valuable. The author proposes specific reformulations for several questions to move from subjective "open" interpretations to more objective categories.

Summary updated

Abstract

Background: Relevance to industry and scientific rigor have long been an area of friction in IS research. However little work has been done on how to evaluate IS research relevance. Kitchenham et al [13] proposed one of the few relevance evaluating instruments in literature, later revised by Daneva et al [7]. Aim: To analyze the practitioner/consultant perspective checklist1 for relevance in order to evaluate its comprehensibility and applicability from the point of view of the practitioner/consultant in the context of an advanced university classroom. Method: Five master level students in the field of IS assessed a set of 24 papers using the relevance checklist1. For each question in the checklist, inter-rater agreement has been calculated and the reasoning that the practitioners applied has been reconstructed from comments. Results: Inter-rater agreement only showed to be slight for three questions and poor for all other questions. Analysis of comments provided by the practitioners showed only two questions that were interpreted in the same way by all practitioners. These two questions showed significantly higher inter-rater agreement than other questions. Conclusions: The generally low inter-rater agreement could be explained as an indication that the checklist1 is in its current form not appropriate for measuring industry relevance of IS research. The different interpretations found for the checklist questions provide useful insight for reformulation of questions. Reformulations are proposed for some questions.
BibTeX
@misc{tax2014analyzing,
  author = {Niek Tax},
  title = {{Analyzing a practitioner perspective on relevance of published empirical research in Requirements Engineering}},
  year = {2014},
  eprint = {1403.1112},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1403.1112},
}
Figure