Abstract
Language workbenches are tools that enable the definition, reuse, and composition of programming languages and their ecosystems, aiming to streamline language development. To facilitate their adoption by language designers, the comprehensibility of the language used to define other languages is an important aspect to evaluate. Moreover, considering that language workbenches are relatively new tools, user acceptance emerges as a crucial factor to be accounted for during their assessment. Current literature often neglects user-centred aspects like comprehensibility and acceptance in the assessment of this breed of tools. This paper addresses this gap through a family of experiments assessing Neverlang, a modular language workbench. The study adopts a tailored version of the Method Evaluation Model (MEM) to evaluate the comprehensibility of Neverlang's meta-language and programs, as well as user acceptance in terms of perceived ease of use, perceived usefulness, and intention to use. It also investigates the relationships among these dimensions. The experiments were conducted in three iterations involving participants from academia. The results reveal that users demonstrate sufficient comprehension of Neverlang's meta-language, particularly concerning its syntax, express a favourable perception of its usefulness, and indicate their intention to use it. However, the results also indicate that Neverlang's ease of use remains a challenge. Additionally, variations in the perceived ease of use and perceived usefulness, whether low or high, influence the users' intention to use the tool. Surprisingly, no significant correlation is found between comprehensibility and user acceptance. Notably, higher comprehensibility of the meta-language does not necessarily translate into greater acceptance, underscoring the complex interplay between comprehension and adoption.
Introduction
Creating a new programming language or a Domain-Specific Language (DSL) from scratch is a complex engineering challenge involving the construction of parsers, compilers, and IDE tooling. Language workbenches like JetBrains MPS, Spoofax, or Neverlang aim to simplify this by providing a "meta-language"—a specialized language used to define other languages modularly.
While technical metrics like feature coverage and execution time are common in literature, the human element is often overlooked. If a meta-language is unintuitive or frustrating, it will fail to gain adoption regardless of its technical power. This study empirically evaluates the modular language workbench Neverlang, focusing on two critical human-centric pillars: comprehensibility (can users understand it?) and user acceptance (do they want to use it?).
Methodology
The researchers conducted a family of experiments consisting of an original study and two independent replications, involving 50 academic participants (undergraduate, master's, and PhD students). The study utilized an adapted version of the Method Evaluation Model (MEM) to bridge the gap between objective performance and subjective perception.
As shown in the model above, the evaluation was split into two main categories:
- Actual Success (Comprehension): Measured through tests assessing learnability (syntax), understandability (semantics), and evolving (code modification).
- Perceived Success (Acceptance): Measured via Likert-scale questionnaires assessing Perceived Ease of Use (PEOU), Perceived Usefulness (PU), and Intention to Use (ITU).
Data was collected across three cohorts: UniMI 2022, BISS 2023, and UniMI 2023. Statistical analysis, including Wilcoxon signed-rank and Spearman's Rank Correlation, was used to aggregate results and identify relationships between these dimensions.
Key Insights
The Comprehension-Acceptance Gap
The most significant finding is that actual comprehension does not predict user acceptance. A user's ability to successfully navigate the comprehension test had no statistical correlation with their desire to use the tool. This suggests that making a tool easier to learn does not automatically make it feel "easy to use" or desirable for long-term adoption.
The Modularity Tax
Neverlang’s core strength is its modularity, allowing languages to be built from reusable slices. However, this paradigm shift introduces a high cognitive load. While participants quickly grasped basic syntax (high learnability), they struggled more with modifying existing programs (evolving). The fragmented nature of modular codebases can be difficult for developers accustomed to monolithic structures to navigate.
Usefulness as a Driver
Despite finding the tool difficult to use, participants overwhelmingly recognized its functional value. This indicates that in specialized developer tooling, high perceived usefulness can drive adoption even when the user experience is challenging. Users are willing to tolerate a steep learning curve if the payoff in productivity is clear.
Results
- High Comprehension: Meta-analysis showed that 76% of users achieved sufficient total comprehension, with syntax learnability peaking at 84%.
- Acceptance Metrics:
- 92% of users rated Neverlang as useful (Perceived Usefulness).
- 62% expressed an intention to use it in the future (Intention to Use).
- Only 48% found the tool easy to use (Perceived Ease of Use).
- Correlations: While Perceived Usefulness and Perceived Ease of Use both strongly correlated with Intention to Use, there was zero significant correlation between actual comprehension scores and any acceptance metrics.
Discussion and Future Work
The study highlights that Neverlang's ease of use remains a primary hurdle. The lack of correlation between comprehension and acceptance suggests that improving the meta-language's documentation or syntax alone may not be enough to drive adoption; the overall user experience and tooling ecosystem must also be addressed.
Future research should include longitudinal studies to see how user perception changes as they become more familiar with modularity over time. Additionally, comparative studies against other workbenches like MPS or Spoofax could provide a benchmark for usability in the field. To improve adoption, the authors suggest developing better IDE support and graphical interfaces to mitigate the complexity of the meta-language.
Summary updated