Abstract
The gap between academic research and industrial practice in Information Systems (IS) remains a significant hurdle. This paper evaluates a checklist designed to measure the practical relevance of research papers, specifically within the domain of Requirements Engineering (RE). Through an empirical study involving five practitioners and 24 research papers, we assess the inter-rater reliability of the checklist and the correlation between academic impact and practical utility. Our findings reveal a "poor" level of agreement among raters due to significant linguistic ambiguity and subjective interpretations. Additionally, we find no statistically significant correlation between citation counts and practitioner relevance scores, suggesting that scientific rigor does not necessarily translate to industry value.
Introduction
In Information Systems, "scientific rigor" often takes precedence over "practical relevance." While various instruments have been proposed to evaluate the utility of research for practitioners, these tools are rarely validated for usability. This study, conducted by Niek Tax, investigates the reliability of a checklist proposed by Kitchenham et al. and revised by Daneva et al. The goal is to determine if practitioners can reach a consensus when using such a tool to rate the relevance of research in Requirements Engineering.
Methods
The study employed five Master’s level students specializing in Computer Science or Business Information Technology to act as "practitioners." These participants possessed significant academic knowledge and some practical exposure to Requirements Engineering. The methodology consisted of three main pillars:
- Evaluation: Each participant applied a 20-question checklist to a set of 24 empirical research papers focused on Software Requirement Specification (SRS) comprehensibility.
- Statistical Analysis: Fleiss’ kappa was calculated to determine the degree of agreement among multiple raters. Additionally, Kendall’s tau measured the correlation between the papers' relevance scores and their scientific impact, determined by citation counts via Scopus.
- Qualitative Reconstruction: By analyzing the comments left by the raters, the author attempted to "reverse-engineer" why different people interpreted the same questions in wildly different ways, identifying specific linguistic and conceptual hurdles.
Results and Discussion
The results highlight a significant "agreement crisis." For 15 out of 19 questions, the agreement was negative, indicating less consensus than random chance. Only two questions reached "slight agreement": Q2 ("Is it claimed that the results are industry-relevant?") and Q5 ("Is it explicitly stated what the implications... for practice are?").
The qualitative data revealed that terms like "believable evidence" were interpreted inconsistently—some raters sought statistical significance, while others relied on intuition. This subjectivity renders the resulting "relevance score" largely meaningless. Furthermore, the checklist's inconsistent use of active and passive voice likely hindered comprehensibility. Drawing on writing style guides, the author suggests this linguistic friction contributed to the poor performance of the tool.
Crucially, the study found no link between academic success and practical utility. The Kendall’s tau coefficient was -0.0988 (p=0.526), indicating that highly cited papers are not necessarily perceived as more relevant by practitioners.
Conclusion
The current checklist for assessing research relevance is hampered by subjectivity and linguistic friction. The findings suggest that the current tool is not fit for purpose in its current form. To bridge the rigor-relevance gap, evaluation tools must move toward objective, nominal categories. Future efforts should focus on reformulating these instruments to ensure consistent results across different evaluators, ultimately helping researchers produce work that is both scientifically sound and industrially valuable. The author proposes specific reformulations for several questions to move from subjective "open" interpretations to more objective categories.
Summary updated