A Retrieval-Augmented Generation Approach to Extracting Algorithmic Logic from Neural Networks

Waleed Khalid, Dmitry Ignatov, Radu TimoftearXiv 2025 · DOI · arXiv

TL;DR

NN-RAG is a retrieval-augmented system that automatically extracts, reconstructs, and validates standalone PyTorch modules from the vast, fragmented world of open-source code. By resolving complex dependencies and verifying executability in a sandbox, it turns messy GitHub repositories into a searchable library of "building blocks" that can be reliably reused by researchers and LLMs alike.

Introduction

Deep learning research is built on the shoulders of giants, but those giants often have very messy codebases. While the PyTorch ecosystem is full of innovative layers, loss functions, and architectures, "porting" a promising component from a random repository into your own project is often a manual nightmare.

Researchers typically face three major friction points: discovery of the right code at the right granularity, "dependency hell" involving transitive imports, and the validation of functional executability. NN-RAG aims to bridge this gap by treating the open-source corpus as a dependable, executable database rather than just a collection of text files.

Methodology

The authors propose a "retrieval-first" stance. Instead of asking a Large Language Model (LLM) to synthesize code from scratch—which risks hallucinations—NN-RAG retrieves real, verified code. The system operates through a rigorous seven-phase pipeline designed to ensure that every extracted block is self-contained and functional.

The seven-phase NN-RAG pipeline for automated block discovery, dependency resolution, and validation.

Key phases include:

  • Block Discovery: Statically scanning repositories for classes inheriting from nn.Module with a forward() method.
  • Scope-Aware Dependency Resolution: Using the LEGB (Local, Enclosing, Global, Built-in) model to analyze Python scopes and resolve imports to ensure the block is "scope-closed."
  • Three-Stage Validation: Every block must pass an AST parse, bytecode compilation, and a sandboxed execution test before being promoted to the library.

Datasets and Implementation

The authors applied NN-RAG to 19 major PyTorch repositories, including heavyweights like torchvision, timm, mmdetection, transformers, and detectron2. The primary target for these extractions was the LEMUR dataset, a benchmark for AutoML and neural architecture discovery. NN-RAG was used to populate LEMUR with a diverse array of novel, executable network structures.

Key Insights

  • The Power of the Long Tail: While marquee libraries provide many blocks, a "long tail" of smaller projects contributed a massive variety of unique architectural ideas.
  • Validation as a Filter: Most failures in the pipeline were not due to syntax errors but rather hidden complexities like dynamic metaprogramming or dependencies on external C++/CUDA kernels.
  • Accidental SOTA: By recombining design ideas surfaced by the pipeline, the authors identified a model that achieved the highest accuracy to date on the LEMUR benchmark, suggesting that the "best" architectures might already exist in pieces across the web.

Quantitative Results

The system demonstrated high efficiency and effectiveness in transforming raw repository data into validated modules. The average extraction time was recorded at roughly 2.5 seconds per block, supported by a 95% cache-hit rate on subsequent runs.

Statistical breakdown of block extraction and validation success rates.
  • Scale: Extracted 1,289 candidate blocks and successfully validated 941 (73%) as executable.
  • Uniqueness: Over 80% of the extracted modules were found to be structurally unique.
  • Contribution: NN-RAG supplied approximately 72% of all novel network structures in the current LEMUR dataset snapshot.
  • Performance: A model assembled via the NN-RAG framework achieved 92.81% accuracy on CIFAR-10, outperforming existing models on the LEMUR leaderboard.

Limitations and Future Work

The authors acknowledge that the current sandbox cannot easily validate modules requiring custom C++ or CUDA toolchains. Additionally, Python's "string-to-class" registries remain partially opaque to static analysis. Future work will focus on per-repository "build recipes" and tighter dependency closure using richer metadata to ensure auxiliary packages are correctly pinned and reproduced.

Summary updated

Abstract

Reusing existing neural-network components is central to research efficiency, yet discovering, extracting, and validating such modules across thousands of open-source repositories remains difficult. We introduce NN-RAG, a retrieval-augmented generation system that converts large, heterogeneous PyTorch codebases into a searchable and executable library of validated neural modules. Unlike conventional code search or clone-detection tools, NN-RAG performs scope-aware dependency resolution, import-preserving reconstruction, and validator-gated promotion -- ensuring that every retrieved block is scope-closed, compilable, and runnable. Applied to 19 major repositories, the pipeline extracted 1,289 candidate blocks, validated 941 (73.0%), and demonstrated that over 80% are structurally unique. Through multi-level de-duplication (exact, lexical, structural), we find that NN-RAG contributes the overwhelming majority of unique architectures to the LEMUR dataset, supplying approximately 72% of all novel network structures. Beyond quantity, NN-RAG uniquely enables cross-repository migration of architectural patterns, automatically identifying reusable modules in one project and regenerating them, dependency-complete, in another context. To our knowledge, no other open-source system provides this capability at scale. The framework's neutral specifications further allow optional integration with language models for synthesis or dataset registration without redistributing third-party code. Overall, NN-RAG transforms fragmented vision code into a reproducible, provenance-tracked substrate for algorithmic discovery, offering a first open-source solution that both quantifies and expands the diversity of executable neural architectures across repositories.
BibTeX
@misc{khalid2025retrieval,
  author = {Waleed Khalid and Dmitry Ignatov and Radu Timofte},
  title = {{A Retrieval-Augmented Generation Approach to Extracting Algorithmic Logic from Neural Networks}},
  year = {2025},
  eprint = {2512.04329},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.04329},
}
Figure