Nalanda: a socio-technical graph platform for building software analytics tools at enterprise scale

Chandra Shekhar Maddila, Suhas Shanbhogue, Apoorva Agrawal, Thomas Zimmermann, Chetan Bansal, Nicole Forsgren, Divyanshu Agrawal, Kim Herzig, Arie van Deursen

TL;DR

This paper introduces Nalanda, a large-scale socio-technical graph platform designed to tackle information overload and discovery challenges in enterprise software development. By integrating data from various sources into a comprehensive graph and index system, Nalanda powers applications like a personalized news feed and intelligent artifact/expert recommenders, demonstrating significant improvements in developer efficiency and information access.

What Problem Does This Paper Address?

Modern software development is a complex, collaborative endeavor involving numerous artifacts (code, work items, pull requests) and stakeholders. As projects scale, the sheer volume of information and the distributed nature of data sources create significant challenges for developers. Finding relevant information, understanding project history, and identifying subject matter experts become time-consuming and difficult tasks, leading to information overload and reduced productivity. Existing tools often focus on specific aspects or operate at a smaller scale, failing to address the enterprise-wide need for integrated, intelligent information discovery.

Methods

Nalanda is built upon two core subsystems:

  • Nalanda Graph System: This subsystem constructs a large-scale socio-technical graph that captures relationships between entities like users, files, repositories, pull requests, and work items. It uses a native graph database and is optimized for querying complex relationships. The graph schema includes nodes for users, files, repositories, pull requests, and work items, connected by edges representing actions like "changes," "creates," "reports to," and "comments on."
  • Nalanda Index System: This subsystem provides enterprise-scale indexing capabilities for efficient search and recommendation. It includes an artifact index (for pull requests and work items) and an expert index (mapping subject matter experts to skills). The BM25 algorithm is used for text similarity and ranking in both indices.

The platform employs a two-pipeline architecture for data collection and graph construction:

  1. Data Aggregation Pipeline: Fetches data from sources like CloudMine using Lens Orchestrator and stores it in Azure Blob Storage.
  2. Graph Construction Pipeline: Processes the aggregated data using Azure Batch jobs to build and update the Nalanda graph in Azure Cosmos DB. This pipeline operates in bootstrap and incremental modes for efficiency.

A "self-healing" system is implemented to detect and remediate data gaps, ensuring data consistency.

Datasets

The Nalanda platform ingests data from 6,500 Microsoft repositories. For evaluation, the paper uses:

  • Nalanda's internal data: This includes data from 6,500 repositories, resulting in a socio-technical graph with 37.4 million nodes and 128.7 million edges, and an artifact index with 8 million documents and an expert index with 61,000 documents.
  • Azure DevOps: Used as the primary source for work items, pull requests, and code reviews.
  • StackOverflow (private instance): Used to create ground truth data for expert recommendations.

The paper does not introduce new benchmark datasets but rather leverages existing data sources within Microsoft.

Key Insights and Ideas

  • The Power of Socio-Technical Graphs at Scale: Nalanda demonstrates that building and leveraging a socio-technical graph at an enterprise scale (thousands of repositories) can effectively bridge the gap between technical artifacts and social interactions in software development.
  • Integrated Information Discovery: By combining a graph representation with indexing, Nalanda provides a unified view of development activities, reducing the need for developers to navigate multiple disparate systems.
  • Developer-Centric News Feed: The MyNalanda application highlights the value of a personalized news feed that aggregates relevant activities from a developer's various projects, improving situational awareness.
  • Contextual Recommendations: The artifact and expert recommendation system, enhanced by the socio-technical graph for re-ranking, significantly improves the accuracy and relevance of suggestions, helping developers find related work and experts more efficiently.
  • Self-Healing for Data Consistency: In a large-scale distributed system, proactive detection and remediation of data gaps are crucial for maintaining data integrity and the reliability of downstream applications.

Key Results

  • MyNalanda Usage: Reached 290 Daily Active Users (DAU) and 590 Monthly Active Users (MAU) within six months of its beta deployment. A survey indicated that 74% of developers and engineering managers found MyNalanda useful for information discovery. Active items (pull requests, code reviews) were rated as the most valuable features.
  • Recommendation System Performance:
    • Incorporating more artifact attributes (title, description) and the socio-technical graph improved artifact recommendation accuracy.
  • The artifact recommendation task saw accuracy improve to 0.78 (a 30.45 percentage point increase).
  • A user study with ten developers indicated that a system like Nalanda could reduce the time spent and the number of places needed to find information.
  • Participants rated artifact recommendations as more useful than expert recommendations, with a notable difference between expected and perceived usefulness for expert recommendations.

Limitations and Future Work

  • Limitations:
    • External Validity: The evaluation and user studies were conducted within a single company (Microsoft). The results may not generalize to all developers in all contexts or to open-source communities.
  • Response Bias: While participants were organizationally distant from the development team, there's a slight possibility of positive bias.
  • Expert Recommendation Perception: Users perceived expert recommendations as less useful than artifact recommendations, with a gap between expectations and actual perception.
  • Future Work:
    • Scaling Nalanda and MyNalanda significantly within Microsoft.
  • Implementing the Nalanda graph platform as a service for open-source data (e.g., from GitHub).
  • Exploring other recommendation types, such as internal/external documentation, tutorials, and Q&A forums.
  • Leveraging advances in deep learning and Graph Convolutional Networks (GCNs) for applications like neural reviewer recommendations.
  • Integrating graph representation techniques like link prediction and social network analysis into software engineering analytics.

Summary updated

Abstract

Software development is information-dense knowledge work that requires collaboration with other developers and awareness of artifacts such as work items, pull requests, and files. With the speed of development increasing, information overload is a challenge for people developing and maintaining these systems. Finding information and people is difficult for software engineers, especially when they work in large software systems or have just recently joined a project. In this paper, we build a large scale data platform named Nalanda platform, which contains two subsystems: 1. A large scale socio-technical graph system, named Nalanda graph system 2. A large scale recommendation system, named Nalanda index system that aims at satisfying the information needs of software developers. The Nalanda graph is an enterprise scale graph with data from 6,500 repositories, with 37,410,706 nodes and 128,745,590 edges. On top of the Nalanda graph system, we built software analytics applications including a newsfeed named MyNalanda, and based on organic growth alone, it has Daily Active Users (DAU) of 290 and Monthly Active Users (MAU) of 590. A preliminary user study shows that 74% of developers and engineering managers surveyed are favorable toward continued use of the platform for information discovery. The Nalanda index system constitutes two indices: artifact index and expert index. It uses the socio-technical graph (Nalanda graph system) to rank the results and provide better recommendations to software developers. A large scale quantitative evaluation shows that the Nalanda index system provides recommendations with an accuracy of 78% for the top three recommendations.
BibTeX
@inproceedings{maddila2022nalanda,
  author = {Chandra Shekhar Maddila and Suhas Shanbhogue and Apoorva Agrawal and Thomas Zimmermann and Chetan Bansal and Nicole Forsgren and Divyanshu Agrawal and Kim Herzig and Arie van Deursen},
  title = {{Nalanda: a socio-technical graph platform for building software analytics tools at enterprise scale}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2022},
  doi = {10.1145/3540250.3558949},
}
Figure