Introduction
Database connectors, such as JDBC or ODBC drivers, serve as critical infrastructure bridging applications and Database Management Systems (DBMS). Despite their importance, testing these components is challenging due to subtle logic, vendor-specific deviations from standards, and complex stateful behaviors. Traditional fuzzing techniques often focus on SQL generation, leaving the internal logic and API handling of connectors largely unexplored. This paper introduces a novel framework that leverages Large Language Models (LLMs) and Reinforcement Learning (RL) to automate the discovery of bugs and non-standard implementations in database connectors through dynamic differential testing.
Methodology: A Closed-Loop Framework
The proposed system employs a closed-loop architecture designed to iteratively improve its bug-finding capabilities. The core process involves five key stages:
- Structured Prompting: The system uses parameterized templates to guide the LLM. These templates define roles (e.g., DBMS testing expert), provide context (schema, prioritized operations), and set design directives for generating "chaotic" yet complete Java test cases.
- RL-Guided Selection: To optimize the search space, the framework treats prompt selection as a Multi-Armed Bandit problem. Using the UCB1 (Upper Confidence Bound) algorithm, it adaptively selects prompt templates that have historically yielded more inconsistencies.
- Differential Testing: Generated test cases are executed against two compatible connectors (e.g., MySQL and OceanBase).
- Feedback Loop: Inconsistencies in execution results or return values are flagged as potential bugs. This signal is fed back to the RL engine to reward successful prompts.
- Property Exploration: The system systematically varies connection properties (e.g.,
rewriteBatchedStatements) to expose edge-case behaviors.
Experimental Evaluation
The framework was evaluated using the MySQL Connector/J (v9.2.0) and OceanBase Connector/J (v4.2.0), with Qwen-plus serving as the test case generator. The primary objective was to identify behavioral discrepancies that indicate either specification violations or implementation bugs.
Key Findings
The automated testing process successfully identified 16 distinct issues, categorized into confirmed bugs and unsafe (non-standard) implementations.
Significant Discoveries
- Long-standing Bugs: The tool uncovered a bug in the MySQL connector (Issue 3, related to
getHoldability()) that had persisted for 17 years. - Specification Violations: Several issues involved connectors returning illegal values instead of throwing mandatory
SQLExceptions, particularly during batch operations or on specificResultSettypes. - Compatibility Trade-offs: Six "unsafe implementations" were found in OceanBase. These were intentional deviations from the JDBC standard designed to maintain compatibility with legacy, erroneous MySQL behaviors, highlighting the complexity of the connector ecosystem.
Conclusion and Future Work
This research demonstrates that LLMs, when guided by RL and differential testing, can effectively explore the complex state space of database connectors. By generating full Java test cases rather than just SQL strings, the framework exposes deep logic errors that traditional tools miss. Future work will focus on expanding the framework to a broader range of DBMS drivers, performing deeper root-cause analysis, and establishing more robust baselines against traditional fuzzing methods.
Summary updated