Nalanda: A Socio-Technical Graph for Building Software Analytics Tools at Enterprise Scale

Chandra Maddila, Suhas Shanbhogue, Apoorva Agrawal, Thomas Zimmermann, Chetan Bansal, Nicole Forsgren, Divyanshu Agrawal, Kim Herzig, Arie van Deursen

TL;DR

Nalanda is a massive socio-technical graph platform built at Microsoft to solve information discovery at enterprise scale. By mapping the complex relationships between 37 million nodes (developers, pull requests, files, and work items), it powers a developer-centric news feed and a recommendation engine that significantly improves the accuracy of finding relevant code artifacts and subject matter experts.

What Problem Does This Paper Address?

In large-scale software engineering, developers are often drowning in data but starving for information. As projects grow to include thousands of repositories and team members, finding the right person to ask for help or locating a relevant code sample becomes a "needle in a haystack" problem.

Traditional tools are often work-item centric, meaning you have to know exactly what you are looking for to find it. Furthermore, knowledge is often siloed across disparate systems (version control, bug trackers, organizational charts). The authors address this by creating a developer-centric platform that integrates these silos into a single, queryable socio-technical graph, helping engineers stay aware of their teammates' work and discover relevant expertise without manual "git history mining."

Methods

The Nalanda platform consists of two primary subsystems:

  1. The Nalanda Graph System: A socio-technical graph stored in a native graph database (Azure CosmosDB). It uses a schema that connects actors (Authors, Reviewers) to technical artifacts (Pull Requests, Files, Repositories, Work Items).
  2. Pipeline: The system uses a two-stage pipeline. A Data Aggregation Pipeline fetches raw events, and a Graph Construction Pipeline processes them into nodes and edges.
  3. Efficiency: To handle the scale, the authors implemented a "Bootstrap" mode for initial onboarding and an "Incremental" mode for daily updates, which reduced pipeline run time by 98.8%.
Figure 1: Nalanda’s Graph Schema
  1. The Nalanda Index System: An enterprise-scale search system built on Azure Cognitive Search.
  2. Ranking Algorithm: The system uses the standard BM25 algorithm for initial text similarity. However, its "secret sauce" is the re-ranking step. It uses the socio-technical graph to calculate the "proximity score" (shortest path distance) between the developer performing the search and the results. Items closer to the developer in the social or technical network are ranked higher.
Figure 5: Nalanda artifact and expert recommendation ap- plication architecture

Datasets

The platform was built and evaluated using internal Microsoft data:

  • Core Graph: Data from 6,500 repositories, resulting in 37,410,706 nodes and 128,745,590 edges.
  • Artifact Evaluation: A random sample of 80,000 work items linked to pull requests served as the ground truth for testing recommendations.
  • Expert Evaluation: A dataset of 10,000 sampled questions and answers from a private internal instance of StackOverflow was used to evaluate the expert recommendation engine.

Key Insights and Ideas

  • Proximity Matters: The most significant insight is that technical relevance (text similarity) is not enough. By incorporating socio-technical proximity—how "close" a developer is to a piece of code or another person in the organizational and activity graph—the system provides much more useful recommendations.
  • Developer-Centric Awareness: The success of the "MyNalanda" news feed (Figure 3) suggests that developers value a centralized "social media style" feed of activity relevant to their specific sphere of influence, rather than hunting through individual repositories.
  • Self-Healing Pipelines: At enterprise scale, data gaps are inevitable due to infrastructure failures. The authors implemented a self-healing mechanism that automatically detects gaps in the incremental stream and triggers a targeted bootstrap to ensure graph consistency.

Key Results

The authors deployed two main applications on top of Nalanda:

  • MyNalanda (News Feed): Reached 290 Daily Active Users (DAU) and 590 Monthly Active Users (MAU) through organic growth alone. 74% of surveyed users were favorable toward continued use.
  • Artifact Recommendation: The inclusion of the socio-technical graph for re-ranking provided a massive boost. Accuracy at K=3 jumped from 0.26 (using only metadata) to 0.71 (using the full graph).
  • Expert Recommendation: The system achieved an accuracy of 0.63 at K=3, helping new developers find the right people to talk to in large, remote teams.
Figure 4: Nalanda recommendations in Azure DevOps

Limitations and Future Work

  • Internal Validity: The study was conducted entirely within Microsoft. While the scale is vast, the results might be influenced by Microsoft's specific engineering culture and tooling.
  • Response Bias: The user study involved a small group (10 developers), and while they were organizationally distant from the authors, a small risk of positive bias remains.
  • Future Directions: The authors plan to explore Graph Convolutional Networks (GCNs) for neural reviewer recommendations and expand the platform to include external documentation and tutorials. They also see potential in offering Nalanda as a service for open-source data (e.g., GitHub).

Summary updated

Abstract

Software development is information-dense knowledge work that requires collaboration with other developers and awareness of artifacts such as work items, pull requests, and files. With the speed of development increasing, information overload is a challenge for people developing and maintaining these systems. Finding information and people is difficult for software engineers, especially when they work in large software systems or have just recently joined a project. In this paper, we build a large scale data platform named Nalanda platform, which contains two subsystems: 1. A large scale socio-technical graph system, named Nalanda graph system 2. A large scale recommendation system, named Nalanda index system that aims at satisfying the information needs of software developers. The Nalanda graph is an enterprise scale graph with data from 6,500 repositories, with 37,410,706 nodes and 128,745,590 edges. On top of the Nalanda graph system, we built software analytics applications including a newsfeed named MyNalanda, and based on organic growth alone, it has Daily Active Users (DAU) of 290 and Monthly Active Users (MAU) of 590. A preliminary user study shows that 74% of developers and engineering managers surveyed are favorable toward continued use of the platform for information discovery. The Nalanda index system constitutes two indices: artifact index and expert index. It uses the socio-technical graph (Nalanda graph system) to rank the results and provide better recommendations to software developers. A large scale quantitative evaluation shows that the Nalanda index system provides recommendations with an accuracy of 78% for the top three recommendations.
BibTeX
@inproceedings{maddila2022nalanda,
  author = {Chandra Maddila and Suhas Shanbhogue and Apoorva Agrawal and Thomas Zimmermann and Chetan Bansal and Nicole Forsgren and Divyanshu Agrawal and Kim Herzig and Arie van Deursen},
  title = {{Nalanda: A Socio-Technical Graph for Building Software Analytics Tools at Enterprise Scale}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2022},
  doi = {10.1145/3540250.3558949},
}
Figure