TL;DR
NN-RAG is a retrieval-augmented system that automatically extracts, reconstructs, and validates standalone PyTorch modules from the vast, fragmented world of open-source code. By resolving complex dependencies and verifying executability in a sandbox, it turns messy GitHub repositories into a searchable library of "building blocks" that can be reliably reused by researchers and LLMs alike.
Introduction
Deep learning research is built on the shoulders of giants, but those giants often have very messy codebases. While the PyTorch ecosystem is full of innovative layers, loss functions, and architectures, "porting" a promising component from a random repository into your own project is often a manual nightmare.
Researchers typically face three major friction points: discovery of the right code at the right granularity, "dependency hell" involving transitive imports, and the validation of functional executability. NN-RAG aims to bridge this gap by treating the open-source corpus as a dependable, executable database rather than just a collection of text files.
Methodology
The authors propose a "retrieval-first" stance. Instead of asking a Large Language Model (LLM) to synthesize code from scratch—which risks hallucinations—NN-RAG retrieves real, verified code. The system operates through a rigorous seven-phase pipeline designed to ensure that every extracted block is self-contained and functional.
Key phases include:
- Block Discovery: Statically scanning repositories for classes inheriting from
nn.Modulewith aforward()method. - Scope-Aware Dependency Resolution: Using the LEGB (Local, Enclosing, Global, Built-in) model to analyze Python scopes and resolve imports to ensure the block is "scope-closed."
- Three-Stage Validation: Every block must pass an AST parse, bytecode compilation, and a sandboxed execution test before being promoted to the library.
Datasets and Implementation
The authors applied NN-RAG to 19 major PyTorch repositories, including heavyweights like torchvision, timm, mmdetection, transformers, and detectron2. The primary target for these extractions was the LEMUR dataset, a benchmark for AutoML and neural architecture discovery. NN-RAG was used to populate LEMUR with a diverse array of novel, executable network structures.
Key Insights
- The Power of the Long Tail: While marquee libraries provide many blocks, a "long tail" of smaller projects contributed a massive variety of unique architectural ideas.
- Validation as a Filter: Most failures in the pipeline were not due to syntax errors but rather hidden complexities like dynamic metaprogramming or dependencies on external C++/CUDA kernels.
- Accidental SOTA: By recombining design ideas surfaced by the pipeline, the authors identified a model that achieved the highest accuracy to date on the LEMUR benchmark, suggesting that the "best" architectures might already exist in pieces across the web.
Quantitative Results
The system demonstrated high efficiency and effectiveness in transforming raw repository data into validated modules. The average extraction time was recorded at roughly 2.5 seconds per block, supported by a 95% cache-hit rate on subsequent runs.
- Scale: Extracted 1,289 candidate blocks and successfully validated 941 (73%) as executable.
- Uniqueness: Over 80% of the extracted modules were found to be structurally unique.
- Contribution: NN-RAG supplied approximately 72% of all novel network structures in the current LEMUR dataset snapshot.
- Performance: A model assembled via the NN-RAG framework achieved 92.81% accuracy on CIFAR-10, outperforming existing models on the LEMUR leaderboard.
Limitations and Future Work
The authors acknowledge that the current sandbox cannot easily validate modules requiring custom C++ or CUDA toolchains. Additionally, Python's "string-to-class" registries remain partially opaque to static analysis. Future work will focus on per-repository "build recipes" and tighter dependency closure using richer metadata to ensure auxiliary packages are correctly pinned and reproduced.
Summary updated