LLM-based Dynamic Differential Testing for Database Connectors with Reinforcement Learning-Guided Prompt Selection

Ce Lyu, Minghao Zhao, Yanhao Wang, Liang Jie

TL;DR

This paper introduces a novel approach to finding bugs in database connectors by using Large Language Models (LLMs) guided by Reinforcement Learning (RL). By dynamically selecting prompts that have historically been good at finding issues, their method effectively generates test cases that uncover subtle vulnerabilities and non-standard behaviors in connectors, leading to the discovery of numerous bugs.

What Problem Does This Paper Address?

Database connectors are the unsung heroes that allow applications to talk to databases. However, they're notoriously tricky to test for vulnerabilities. Unlike typical software bugs, connector flaws often manifest as subtle behavioral quirks or arise from non-standard implementations, making them hard for traditional fuzzing tools to catch. Even advanced LLM-based testing struggles because they lack the specific domain knowledge needed to craft test cases that thoroughly exercise a connector's intricate logic and interfaces. This gap leaves critical components vulnerable, potentially leading to data loss, silent errors, or security breaches.

Methods

The core of this work is an LLM-based test case generation framework enhanced with Reinforcement Learning for prompt selection. Here's how it breaks down:

  1. Parameterized Prompt Template: To imbue the LLM with domain knowledge, the authors designed a structured prompt template. This template guides the LLM to act as an expert in JDBC and database connector testing, covering aspects like role definition, dynamic context (e.g., connection URLs, schemas), task decomposition, and output requirements.
  2. RL-Guided Prompt Selection: Instead of using static prompts, the system employs a Reinforcement Learning approach, specifically inspired by the multi-armed bandit (MAB) framework. Each prompt template is treated as an "arm." The RL agent learns which prompts are most effective at generating test cases that reveal inconsistencies between different connectors. It uses a feedback loop where observed discrepancies (bugs) act as rewards to update the selection strategy, prioritizing prompts that have a higher likelihood of finding issues.
  3. Dynamic Differential Testing: The LLM generates test cases based on the selected prompt. These test cases are then executed against two compatible database connectors (e.g., MySQL Connector/J and OceanBase Connector/J). The results are compared using differential testing. Any detected mismatches between the connectors' behaviors signal a potential bug or unsafe implementation.
  4. Connection Property Exploration: The authors acknowledge that connector behavior is highly sensitive to connection properties (e.g., allowMultiQueries). Their framework systematically explores a curated set of these properties to uncover variations in execution paths and behaviors that might otherwise be missed.

Datasets

The paper does not explicitly mention training datasets for the LLM. The evaluation was performed on two widely used JDBC connectors:

  • MySQL Connector/J: Version 9.2.0, tested against MySQL 8.0.36.
  • OceanBase Connector/J: Version 4.2.0, tested against OceanBase Client 4.2.0 with OceanBase CE v4.2.1.10.

The LLM used for test case generation was Qwen-plus.

Key Insights and Ideas

  • LLMs need domain-specific guidance: Simply asking an LLM to generate test cases isn't enough. A structured, parameterized prompt template is crucial for equipping the LLM with the necessary context and expertise to create effective tests for complex software like database connectors.
  • Reinforcement Learning is key for adaptive testing: Traditional fuzzing or LLM testing can get stuck in local optima. RL allows the testing process to adaptively learn which prompts are most fruitful, focusing computational effort on areas most likely to yield bugs. This is a significant step beyond static prompt generation.
  • Differential testing is powerful for connectors: By comparing the behavior of two different connectors on the same test case, subtle deviations that might go unnoticed by a single connector's internal checks can be readily exposed. This highlights the value of testing against multiple implementations.
  • Connection properties are a hidden bug source: The sensitivity of connectors to various connection properties is a critical factor that many testing approaches overlook. Exploring these properties systematically is essential for comprehensive testing.
  • Long-standing bugs still exist: The discovery of a bug in MySQL Connector/J that has been unfixed for 17 years underscores the difficulty of finding certain types of vulnerabilities and the ongoing need for effective testing tools.

Key Results

The framework was evaluated on MySQL Connector/J and OceanBase Connector/J, leading to the discovery of a total of 16 issues:

  • 7 bugs in MySQL Connector/J.
  • 3 bugs and 6 unsafe implementations in OceanBase Connector/J.

Of these, 10 bugs were officially confirmed, and the remaining were acknowledged as unsafe implementations. The paper highlights two case studies: one where OceanBase Connector/J failed to throw an expected exception for a beforeFirst() call on a forward-only result set (an unsafe implementation), and another where the allowMultiQueries property in MySQL Connector/J caused inconsistent results during batch inserts of duplicate primary keys.

Limitations and Future Work

  • Limitations: The paper presents this as an "early-stage study." While the results are promising, a more comprehensive performance review and baseline comparison are needed. The current study focuses on two specific JDBC connectors, and the generalizability to other connector types or database systems is not fully explored.
  • Future Work: The authors plan to conduct a more in-depth study with a broader performance review, baseline comparisons, and a deeper analysis of the discovered vulnerabilities. They also aim to explore the framework's effectiveness on a wider range of database connectors and systems.

Summary updated

Abstract

Database connectors are critical components enabling applications to interact with underlying database management systems (DBMS), yet their security vulnerabilities often remain overlooked. Unlike traditional software defects, connector vulnerabilities exhibit subtle behavioral patterns and are inherently challenging to detect. Besides, nonstandardized implementation of connectors leaves potential risks (a.k.a. unsafe implementations) but is more elusive. As a result, traditional fuzzing methods are incapable of finding such vulnerabilities. Even for LLM-enable test case generation, due to a lack of domain knowledge, they are also incapable of generating test cases that invoke all interface and internal logic of connectors. In this paper, we propose reinforcement learning (RL)-guided LLM test-case generation for database connector testing. Specifically, to equip the LLM with sufficient and appropriate domain knowledge, a parameterized prompt template is composed which can be utilized to generate numerous prompts. Test cases are generated via LLM with a prompt, and are dynamically evaluated through differential testing across multiple connectors. The testing is iteratively conducted, with each round RL is adopted to select optimal prompt based on prior-round behavioral feedback, so as to maximize control flow coverage. We implement aforementioned methodology in a practical tool and evaluate it on two widely used JDBC connectors: MySQL Connector/J and OceanBase Connector/J. In total, we reported 16 bugs, among them 10 are officially confirmed and the rest are acknowledged as unsafe implementations.
BibTeX
@inproceedings{lyu2025llm,
  author = {Ce Lyu and Minghao Zhao and Yanhao Wang and Liang Jie},
  title = {{LLM-based Dynamic Differential Testing for Database Connectors with Reinforcement Learning-Guided Prompt Selection}},
  booktitle = {ASE},
  year = {2025},
  url = {https://openreview.net/forum?id=WfsXeeVE9R},
}
Figure