TL;DR
This paper explores how better understanding of user needs can lead to more effective software architecture documentation. By comparing two ontologies for knowledge retrieval – one based on expected needs and another on actual needs – the study demonstrates that tailoring ontologies to specific user queries significantly improves the efficiency and effectiveness of finding architectural knowledge.
What Problem Does This Paper Address?
Software architecture documentation is crucial for communicating complex system designs. However, a common problem is that this documentation is often difficult for users to navigate and find the specific information they need. This "one-size-fits-all" approach, where documentation isn't tailored to individual user tasks or questions, leads to wasted time, errors, and reduced value of the documentation itself. The authors investigate whether using ontologies – formal domain models that organize knowledge – can alleviate this problem, and specifically, whether an ontology built with a deep understanding of actual user needs performs better than one built on expected needs.
Methods
The core of this research is a controlled experiment involving master's students in a software engineering course. Participants were tasked with answering specific questions about a software architecture using a semantic wiki called "ArchiMind." The key innovation lies in the comparison of two ontologies used to structure the architectural knowledge within ArchiMind:
- Expected-needs Ontology: This ontology was built based on the authors' understanding of how the software architecture document was organized and what they assumed users might need for an architectural review.
- Actual-needs Ontology: This ontology was constructed by analyzing a set of architectural review questions (from the SEI architectural review approach) and identifying the specific knowledge retrieval needs expressed by those questions.
The experiment measured participants' time-efficiency (how long it took to find answers) and effectiveness (using precision and recall, combined into an F1 score) for each question. Statistical tests (Mann-Whitney-Wilcoxon) were used to compare the performance between the two groups using different ontologies. The authors also analyzed search logs to understand how participants interacted with the "fitting AK organisation" – parts of the ontology that directly supported the answers to specific questions.
Datasets
The primary dataset used was a file-based Software Architecture (SA) document created by students for a social network system. This document comprised 39 pages, 16 diagrams, 528 paragraphs, and over 10,000 words. The content of this SA document was imported into the ArchiMind semantic wiki. The SEI architectural review questions were used to derive the "actual-needs" ontology.
Key Insights and Ideas
- Understanding Actual Needs is Key: The most significant finding is that an ontology built from a deep understanding of actual user needs (derived from specific questions) leads to demonstrably better knowledge retrieval than one based on expected needs.
- "Fitting AK Organisation" Matters: The study highlights the importance of "fitting AK organisation" – ontology elements (classes and relationships) that directly map to the information needed to answer a specific question. When such fitting organization was present, it significantly improved retrieval efficiency and effectiveness.
- Ontology Design Criteria vs. User Needs: The authors found a trade-off. While established ontology design criteria (clarity, coherence, extendibility, minimal encoding bias, minimal ontological commitment) are crucial for creating robust and reusable ontologies, strictly adhering to them sometimes limited the inclusion of "fitting AK organisation" that would have directly supported user needs.
- Redundancy Doesn't Help: Adding fitting AK organization that is already covered by existing, well-structured parts of the ontology does not yield further improvements in retrieval. The benefit comes from filling gaps, not duplicating existing support.
- Traceability is Enhanced: Ontology-based documentation, by explicitly modeling relationships between architectural knowledge (AK) elements, inherently improves traceability, which is a known pain point in traditional documentation.
Key Results
- For Question 3 (architectural patterns), the actual-needs ontology led to significantly higher efficiency (129 seconds faster on average) and effectiveness (0.56 higher F1 score).
- For Question 4 (functional vs. non-functional requirements), the actual-needs ontology showed significantly higher effectiveness (0.67 higher F1 score).
- A strong positive correlation (r = 0.509, p=0.01) was found between the usage of "fitting AK organisation" and "Time-effectiveness" (a combined metric of efficiency and effectiveness). This confirms that actively using the tailored parts of the ontology leads to better results.
- For questions where the fitting AK organization was similar or redundant between the two ontologies (Questions 1, 2, and 5), the differences in efficiency and effectiveness were not statistically significant.
Limitations and Future Work
- Generalizability of Questions: The specific SEI architectural review questions used limit the direct generalizability of findings. Future work could explore a wider range of questions and tasks.
- Generalizability of Documentation: The SA document was student-generated using specific guidelines. While the authors suggest it's comparable to industry practice, further validation with real-world industry documentation would be beneficial.
- Ontology Design Trade-offs: The paper acknowledges the tension between adhering to strict ontology design principles and maximizing "fitting AK organisation." Future work could explore how to better balance these competing concerns.
- More Fitting AK Organisation: The authors plan to investigate how to provide more fitting AK organization for a broader range of user needs while still respecting knowledge-sharing ontology design criteria.
Summary updated