TL;DR
This paper introduces Nalanda, a large-scale socio-technical graph platform designed to tackle information overload and discovery challenges in enterprise software development. By integrating data from various sources into a comprehensive graph and index system, Nalanda powers applications like a personalized news feed and intelligent artifact/expert recommenders, demonstrating significant improvements in developer efficiency and information access.
What Problem Does This Paper Address?
Modern software development is a complex, collaborative endeavor involving numerous artifacts (code, work items, pull requests) and stakeholders. As projects scale, the sheer volume of information and the distributed nature of data sources create significant challenges for developers. Finding relevant information, understanding project history, and identifying subject matter experts become time-consuming and difficult tasks, leading to information overload and reduced productivity. Existing tools often focus on specific aspects or operate at a smaller scale, failing to address the enterprise-wide need for integrated, intelligent information discovery.
Methods
Nalanda is built upon two core subsystems:
- Nalanda Graph System: This subsystem constructs a large-scale socio-technical graph that captures relationships between entities like users, files, repositories, pull requests, and work items. It uses a native graph database and is optimized for querying complex relationships. The graph schema includes nodes for users, files, repositories, pull requests, and work items, connected by edges representing actions like "changes," "creates," "reports to," and "comments on."
- Nalanda Index System: This subsystem provides enterprise-scale indexing capabilities for efficient search and recommendation. It includes an artifact index (for pull requests and work items) and an expert index (mapping subject matter experts to skills). The BM25 algorithm is used for text similarity and ranking in both indices.
The platform employs a two-pipeline architecture for data collection and graph construction:
- Data Aggregation Pipeline: Fetches data from sources like CloudMine using Lens Orchestrator and stores it in Azure Blob Storage.
- Graph Construction Pipeline: Processes the aggregated data using Azure Batch jobs to build and update the Nalanda graph in Azure Cosmos DB. This pipeline operates in bootstrap and incremental modes for efficiency.
A "self-healing" system is implemented to detect and remediate data gaps, ensuring data consistency.
Datasets
The Nalanda platform ingests data from 6,500 Microsoft repositories. For evaluation, the paper uses:
- Nalanda's internal data: This includes data from 6,500 repositories, resulting in a socio-technical graph with 37.4 million nodes and 128.7 million edges, and an artifact index with 8 million documents and an expert index with 61,000 documents.
- Azure DevOps: Used as the primary source for work items, pull requests, and code reviews.
- StackOverflow (private instance): Used to create ground truth data for expert recommendations.
The paper does not introduce new benchmark datasets but rather leverages existing data sources within Microsoft.
Key Insights and Ideas
- The Power of Socio-Technical Graphs at Scale: Nalanda demonstrates that building and leveraging a socio-technical graph at an enterprise scale (thousands of repositories) can effectively bridge the gap between technical artifacts and social interactions in software development.
- Integrated Information Discovery: By combining a graph representation with indexing, Nalanda provides a unified view of development activities, reducing the need for developers to navigate multiple disparate systems.
- Developer-Centric News Feed: The MyNalanda application highlights the value of a personalized news feed that aggregates relevant activities from a developer's various projects, improving situational awareness.
- Contextual Recommendations: The artifact and expert recommendation system, enhanced by the socio-technical graph for re-ranking, significantly improves the accuracy and relevance of suggestions, helping developers find related work and experts more efficiently.
- Self-Healing for Data Consistency: In a large-scale distributed system, proactive detection and remediation of data gaps are crucial for maintaining data integrity and the reliability of downstream applications.
Key Results
- MyNalanda Usage: Reached 290 Daily Active Users (DAU) and 590 Monthly Active Users (MAU) within six months of its beta deployment. A survey indicated that 74% of developers and engineering managers found MyNalanda useful for information discovery. Active items (pull requests, code reviews) were rated as the most valuable features.
- Recommendation System Performance:
- Incorporating more artifact attributes (title, description) and the socio-technical graph improved artifact recommendation accuracy.
- The artifact recommendation task saw accuracy improve to 0.78 (a 30.45 percentage point increase).
- A user study with ten developers indicated that a system like Nalanda could reduce the time spent and the number of places needed to find information.
- Participants rated artifact recommendations as more useful than expert recommendations, with a notable difference between expected and perceived usefulness for expert recommendations.
Limitations and Future Work
- Limitations:
- External Validity: The evaluation and user studies were conducted within a single company (Microsoft). The results may not generalize to all developers in all contexts or to open-source communities.
- Response Bias: While participants were organizationally distant from the development team, there's a slight possibility of positive bias.
- Expert Recommendation Perception: Users perceived expert recommendations as less useful than artifact recommendations, with a gap between expectations and actual perception.
- Future Work:
- Scaling Nalanda and MyNalanda significantly within Microsoft.
- Implementing the Nalanda graph platform as a service for open-source data (e.g., from GitHub).
- Exploring other recommendation types, such as internal/external documentation, tutorials, and Q&A forums.
- Leveraging advances in deep learning and Graph Convolutional Networks (GCNs) for applications like neural reviewer recommendations.
- Integrating graph representation techniques like link prediction and social network analysis into software engineering analytics.
Summary updated