TL;DR
Nalanda is a massive socio-technical graph platform built at Microsoft to solve information discovery at enterprise scale. By mapping the complex relationships between 37 million nodes (developers, pull requests, files, and work items), it powers a developer-centric news feed and a recommendation engine that significantly improves the accuracy of finding relevant code artifacts and subject matter experts.
What Problem Does This Paper Address?
In large-scale software engineering, developers are often drowning in data but starving for information. As projects grow to include thousands of repositories and team members, finding the right person to ask for help or locating a relevant code sample becomes a "needle in a haystack" problem.
Traditional tools are often work-item centric, meaning you have to know exactly what you are looking for to find it. Furthermore, knowledge is often siloed across disparate systems (version control, bug trackers, organizational charts). The authors address this by creating a developer-centric platform that integrates these silos into a single, queryable socio-technical graph, helping engineers stay aware of their teammates' work and discover relevant expertise without manual "git history mining."
Methods
The Nalanda platform consists of two primary subsystems:
- The Nalanda Graph System: A socio-technical graph stored in a native graph database (Azure CosmosDB). It uses a schema that connects actors (Authors, Reviewers) to technical artifacts (Pull Requests, Files, Repositories, Work Items).
- Pipeline: The system uses a two-stage pipeline. A Data Aggregation Pipeline fetches raw events, and a Graph Construction Pipeline processes them into nodes and edges.
- Efficiency: To handle the scale, the authors implemented a "Bootstrap" mode for initial onboarding and an "Incremental" mode for daily updates, which reduced pipeline run time by 98.8%.
- The Nalanda Index System: An enterprise-scale search system built on Azure Cognitive Search.
- Ranking Algorithm: The system uses the standard BM25 algorithm for initial text similarity. However, its "secret sauce" is the re-ranking step. It uses the socio-technical graph to calculate the "proximity score" (shortest path distance) between the developer performing the search and the results. Items closer to the developer in the social or technical network are ranked higher.
Datasets
The platform was built and evaluated using internal Microsoft data:
- Core Graph: Data from 6,500 repositories, resulting in 37,410,706 nodes and 128,745,590 edges.
- Artifact Evaluation: A random sample of 80,000 work items linked to pull requests served as the ground truth for testing recommendations.
- Expert Evaluation: A dataset of 10,000 sampled questions and answers from a private internal instance of StackOverflow was used to evaluate the expert recommendation engine.
Key Insights and Ideas
- Proximity Matters: The most significant insight is that technical relevance (text similarity) is not enough. By incorporating socio-technical proximity—how "close" a developer is to a piece of code or another person in the organizational and activity graph—the system provides much more useful recommendations.
- Developer-Centric Awareness: The success of the "MyNalanda" news feed (Figure 3) suggests that developers value a centralized "social media style" feed of activity relevant to their specific sphere of influence, rather than hunting through individual repositories.
- Self-Healing Pipelines: At enterprise scale, data gaps are inevitable due to infrastructure failures. The authors implemented a self-healing mechanism that automatically detects gaps in the incremental stream and triggers a targeted bootstrap to ensure graph consistency.
Key Results
The authors deployed two main applications on top of Nalanda:
- MyNalanda (News Feed): Reached 290 Daily Active Users (DAU) and 590 Monthly Active Users (MAU) through organic growth alone. 74% of surveyed users were favorable toward continued use.
- Artifact Recommendation: The inclusion of the socio-technical graph for re-ranking provided a massive boost. Accuracy at K=3 jumped from 0.26 (using only metadata) to 0.71 (using the full graph).
- Expert Recommendation: The system achieved an accuracy of 0.63 at K=3, helping new developers find the right people to talk to in large, remote teams.
Limitations and Future Work
- Internal Validity: The study was conducted entirely within Microsoft. While the scale is vast, the results might be influenced by Microsoft's specific engineering culture and tooling.
- Response Bias: The user study involved a small group (10 developers), and while they were organizationally distant from the authors, a small risk of positive bias remains.
- Future Directions: The authors plan to explore Graph Convolutional Networks (GCNs) for neural reviewer recommendations and expand the platform to include external documentation and tutorials. They also see potential in offering Nalanda as a service for open-source data (e.g., GitHub).
Summary updated