Learning to Correct for QA Reasoning with Black-box LLMs

Jaehyung Kim, Dongyoung Kim, Yiming Yang

Abstract

An open challenge in recent machine learning is about how to improve the reasoning capability of large language models (LLMs) in a black-box setting, i.e., without access to detailed information such as output token probabilities. Existing approaches either rely on accessibility (which is often unrealistic) or involve significantly increased train- and inference-time costs. This paper addresses those limitations or shortcomings by proposing a novel approach, namely CoBB (Correct for improving QA reasoning of Black-Box LLMs). It uses a trained adaptation model to perform a seq2seq mapping from the often-imperfect reasonings of the original black-box LLM to the correct or improved reasonings. Specifically, the adaptation model is initialized with a relatively small open-source LLM and adapted over a collection of sub-sampled training pairs. To select the representative pairs of correct and incorrect reasonings, we formulated the dataset construction as an optimization problem that minimizes the statistical divergence between the sampled subset and the entire collection, and solved it via a genetic algorithm. We then train the adaptation model over the sampled pairs by contrasting the likelihoods of correct and incorrect reasonings. Our experimental results demonstrate that CoBB significantly improves reasoning accuracy across various QA benchmarks, compared to the best-performing adaptation baselines.

Introduction

Large Language Models (LLMs) frequently struggle with multi-step reasoning tasks, often producing logical hallucinations despite their scale. While open-source models can be fine-tuned to mitigate these issues, proprietary "black-box" models (e.g., GPT-4, Claude-3) present a significant challenge as their internal weights and output token probabilities are often inaccessible. Existing adaptation strategies typically fall into two categories: prompt engineering, which often hits a performance ceiling, and verifier-based search, which is computationally prohibitive due to high API costs and inference latency.

Comparison of black-box adaptation strategies: (a) probability-based, (b) verifier-based, and (c) the proposed COBB mapping approach.

Methodology

The proposed COBB framework introduces a cost-efficient, model-agnostic correction mechanism. Instead of acting as a standalone solver, the adaptation model acts as a seq2seq mapper that takes the question and the black-box's flawed reasoning as input to output a refined version. The process involves three main phases:

  1. Reasoning Collection: Multiple Chain-of-Thought (CoT) reasonings are sampled from the black-box LLM and labeled as correct or incorrect based on ground-truth answers.
  2. Genetic Algorithm Optimization: To avoid redundancy and computational overhead, dataset construction is treated as an optimization problem. A genetic algorithm selects a representative subset of reasoning pairs by minimizing the 2-Wasserstein distance between the subset and the full collection.
  3. Contrastive Training: A small open-source model (e.g., Mistral-7B) is trained using Odds Ratio Preference Optimization (ORPO). This objective encourages the generation of correct reasoning while explicitly penalizing the likelihood of the black-box's original flawed output.
The COBB framework: reasoning collection, genetic algorithm-based subsampling, and contrastive training of the adaptation model.

The system was evaluated on StrategyQA, GSM8K, TruthfulQA, and ScienceQA benchmarks.

Results

COBB achieved an average accuracy improvement of 6.2% for GPT-3.5-turbo, outperforming the previous state-of-the-art (BBox-Adapter) by 2.2%. Notably, the method is highly efficient, reducing training costs by approximately 80% and inference time by 93% compared to verifier-based baselines. The adaptation model acts as a universal "editor," showing strong transferability across different black-box models like Claude-3 and Llama-3, suggesting that LLMs share common logical failure modes.

Discussion and Conclusion

While COBB significantly enhances black-box reasoning, it remains dependent on ground-truth data for training, limiting its immediate application to purely subjective tasks. Future work may explore parameter-efficient fine-tuning (PEFT) and distillation to further reduce the footprint of the correction module. Overall, COBB demonstrates that a small, well-trained "editor" model can effectively rectify the reasoning errors of much larger proprietary systems.

Summary updated

Abstract

An open challenge in recent machine learning is about how to improve the reasoning capability of large language models (LLMs) in a black-box setting, i.e., without access to detailed information such as output token probabilities. Existing approaches either rely on accessibility (which is often unrealistic) or involve significantly increased train- and inference-time costs. This paper addresses those limitations or shortcomings by proposing a novel approach, namely CoBB (Correct for improving QA reasoning of Black-Box LLMs). It uses a trained adaptation model to perform a seq2seq mapping from the often-imperfect reasonings of the original black-box LLM to the correct or improved reasonings. Specifically, the adaptation model is initialized with a relatively small open-source LLM and adapted over a collection of sub-sampled training pairs. To select the representative pairs of correct and incorrect reasonings, we formulated the dataset construction as an optimization problem that minimizes the statistical divergence between the sampled subset and the entire collection, and solved it via a genetic algorithm. We then train the adaptation model over the sampled pairs by contrasting the likelihoods of correct and incorrect reasonings. Our experimental results demonstrate that CoBB significantly improves reasoning accuracy across various QA benchmarks, compared to the best-performing adaptation baselines.
BibTeX
@inproceedings{kim2024learning,
  author = {Jaehyung Kim and Dongyoung Kim and Yiming Yang},
  title = {{Learning to Correct for QA Reasoning with Black-box LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=8nduk8HMo7},
}
Figure