Abstract
Deep-learning (DL) libraries such as PyTorch and TensorFlow are foundational to modern AI, yet their complexity makes them notoriously difficult to test. Traditional fuzzing techniques often struggle with the "input generation problem" and the "oracle problem." This paper presents DeepREL, a novel fuzzing framework that automatically identifies relational APIs to expand test coverage. By leveraging Natural Language Processing (NLP) to find functionally similar APIs and synthesizing code to map inputs between them, DeepREL successfully tested 157% more APIs than previous state-of-the-art tools. The framework uncovered 162 bugs in PyTorch and TensorFlow, including 23 high-priority issues, demonstrating the efficacy of using library redundancy as a testing resource.
Introduction
As deep learning becomes increasingly integrated into critical infrastructure, the reliability of libraries like TensorFlow and PyTorch is paramount. However, two primary challenges hinder automated testing:
- The Input Generation Problem: Creating valid, high-dimensional tensors that satisfy complex API constraints is difficult. Randomly generated inputs often trigger trivial errors, failing to reach deep logic.
- The Oracle Problem: Even when an input is valid, determining whether the output of a complex mathematical operation is correct is non-trivial without a reference implementation.
Previous state-of-the-art fuzzers, such as FreeFuzz, addressed the input problem by mining examples from documentation and existing test suites. While effective, this approach is limited by the "long-tail" of APIs; if an API lacks documented examples or test cases, it remains unvalidated. DeepREL addresses this gap by automatically inferring relationships between well-documented APIs and their undocumented "siblings."
Methodology
DeepREL introduces Automated Relational API Inference, a technique based on the observation that DL libraries often contain redundant or similar APIs (e.g., aliases, legacy versions, or mathematical inverses). The framework operates in four distinct phases:
1. API Matcher
The system uses NLP to identify candidate API pairs. It calculates Signature Similarity using TF-IDF on function names and argument lists, and Document Similarity using Sentence-BERT on docstrings. This allows DeepREL to find APIs that likely perform similar operations even if their names are not identical.
2. Invocation Synthesizer
Once a candidate pair is identified (e.g., torch.vsplit and torch.tensor_split), DeepREL must determine how to map arguments from the source API to the target API. This is modeled as a maximum weighted bipartite matching problem, allowing the system to automatically generate the necessary code to invoke the target API using inputs originally intended for the source.
3. API Match Verifier
Candidate pairs are validated against a set of real-world inputs to confirm a relationship exists. DeepREL looks for two types of equivalence:
- Value Equivalence: Both APIs produce identical output tensors.
- Status Equivalence: Both APIs behave consistently (e.g., both succeed or both throw the same exception type), which is useful for APIs that perform similar logic but return different formats.
4. API Fuzzer
Verified pairs undergo mutation-based fuzzing. If a pair previously established as equivalent suddenly produces divergent results or crashes, the system flags a potential bug. This relational approach provides a built-in oracle: the established API serves as the ground truth for the target API.
Evaluation and Results
The authors evaluated DeepREL on the latest stable versions of PyTorch (1.10) and TensorFlow (2.7), utilizing a database of inputs mined from documentation, test suites, and over 200 popular DL models.
Coverage Expansion
DeepREL significantly outperformed previous tools by testing 2,973 APIs, a 157% increase over FreeFuzz's 1,158 APIs. By "borrowing" inputs from documented functions, DeepREL successfully reached the long-tail of the API universe that was previously considered untestable.
Bug Discovery
The framework detected 162 total bugs. At the time of the study's conclusion, 106 had been confirmed by developers and 29 were fixed. Notably, DeepREL identified 23 high-priority bugs in PyTorch, representing approximately 13.5% of all high-priority issues reported during the testing window. The tool also identified 14 documentation bugs where the implementation deviated from the official manual.
Discussion and Limitations
While highly effective, DeepREL is not without limitations. The tool reported a false positive rate of approximately 30.7%, typically occurring when the initial verification phase used insufficiently diverse inputs to distinguish between subtly different APIs. Furthermore, the current implementation is limited to 1-to-1 API mappings; it cannot yet infer relationships between a single API and a sequence of multiple APIs (e.g., verifying that a single complex operation matches a combination of simpler ones).
Conclusion
DeepREL demonstrates that the inherent redundancy in modern DL libraries is a powerful asset for automated testing. By treating existing, well-tested APIs as a source of both inputs and expected behavior, DeepREL provides a scalable solution to the coverage and oracle problems in fuzzing. This research highlights a path forward for maintaining the reliability of the rapidly evolving software stacks that power modern artificial intelligence.
Summary updated