Automatic Model Selection with Large Language Models for Reasoning

Xu Zhao, Yuxi Xie, Kenji Kawaguchi, Junxian He, Qizhe Xie

TL;DR

This paper introduces a novel method for automatically selecting between two powerful reasoning approaches, Chain-of-Thought (CoT) and Program-Aided Language Models (PAL), using a Large Language Model (LLM). This approach significantly boosts performance on reasoning tasks, achieving new state-of-the-art results on benchmarks like GSM8K and SVAMP, while also reducing computational costs.

What Problem Does This Paper Address?

Large Language Models (LLMs) have become incredibly adept at various tasks, often by employing a single, unified approach. However, complex reasoning problems can benefit from specialized methods. Two prominent methods are Chain-of-Thought (CoT), which uses natural language for step-by-step reasoning, offering flexibility and interpretability, and Program-Aided Language Models (PAL), which leverage Python code for more structured and rigorous logic. The challenge lies in dynamically choosing the best method for a given problem without knowing the ground truth beforehand. This paper addresses this by proposing an LLM-based model selection strategy to combine the strengths of CoT and PAL.

Methods

The core of this paper is a model selection framework that uses an LLM to dynamically choose between CoT and PAL for a given reasoning problem. The process involves:

  1. Solution Generation: For a given question, an LLM generates both a CoT reasoning chain and a PAL program.
  2. Model Selection: If the outputs from CoT and PAL differ, another LLM (acting as the selector) is prompted with the question and both solutions. The selector LLM is tasked with choosing the better method and providing a brief explanation for its choice. This selection is done using in-context learning, leveraging few-shot examples.

The authors also provide a theoretical analysis to understand the conditions under which this method is effective, focusing on the difference between the models' performance and the accuracy of the selection process.

Datasets

The experiments were conducted on eight reasoning datasets:

  • Arithmetic Reasoning: GSM8K, SVAMP, ASDIV, SingleOP, SingleEQ, AddSub, MultiArith.
  • Symbolic Reasoning: Date Understanding.

The paper does not introduce new benchmarks but utilizes established ones to evaluate the proposed method.

Key Insights and Ideas

  • LLMs as Intelligent Selectors: LLMs, with their in-context learning capabilities, can effectively act as meta-reasoners, selecting the most appropriate reasoning strategy (CoT vs. PAL) for a given problem.
  • Complementarity of CoT and PAL: CoT excels in flexibility and semantic understanding, while PAL offers rigor and computational precision. Combining them through intelligent selection unlocks performance gains beyond either method alone.
  • Theoretical Justification: The paper provides a theoretical framework showing that performance improvements are possible even with imperfect selection, especially when the base models have significantly different error patterns.
  • Efficiency Gains: The proposed method not only improves accuracy but also significantly reduces computation costs, particularly when combined with self-consistency techniques.
  • Broad Applicability: The method demonstrates effectiveness across various backbone LLMs (Codex, ChatGPT, GPT-4, Llama 2) and different reasoning tasks.

Key Results

  • State-of-the-Art Performance: The method achieved new state-of-the-art accuracies of 96.8% on GSM8K and 93.7% on SVAMP.
  • Significant Accuracy Improvements: Across eight reasoning datasets, the proposed model selection consistently improved performance when using Codex, ChatGPT, and GPT-4 as backbone LLMs. For example, with GPT-4, an accuracy of 95.6% on GSM8K and 93.7% on SVAMP was achieved without self-consistency.
  • Reduced Computation Costs: When integrated with self-consistency, the method achieved comparable or better performance than traditional self-consistency approaches while demanding less computation and cost. For instance, using only 5 paths, the method achieved comparable performance to CoT@80 at only 16% of the expense.
  • Robustness: The method showed improvements even when one of the base models performed significantly worse than the other, highlighting its robustness.
  • Effective Explanation Generation: LLMs can not only select the correct model but also provide valid explanations for their choices, especially when they possess strong reasoning capabilities (e.g., GPT-4).

Limitations and Future Work

  • LLM Bias: The method is affected by the inherent biases of LLMs, particularly the order of options presented to the selector LLM. The authors encourage research into bias mitigation strategies.
  • Limited Base Models: The current work focuses on combining only CoT and PAL. Future work could explore incorporating more diverse base models with distinct strengths.
  • Domain Expansion: While successful in reasoning tasks, the authors suggest extending this model selection framework to other domains.
  • Diverse System Instructions: Exploring the use of different system instructions to elicit varied model behaviors for model combinations is a promising avenue.

Summary updated

Abstract

Chain-of-Thought and Program-Aided Language Models represent two distinct reasoning methods, each with its own strengths and weaknesses. We demonstrate that it is possible to combine the best of both worlds by using different models for different problems, employing a large language model (LLM) to perform model selection. Through a theoretical analysis, we discover that the performance improvement is determined by the differences between the combined methods and the success rate of choosing the correct model. On eight reasoning datasets, our proposed approach shows significant improvements. Furthermore, we achieve new state-of-the-art results on GSM8K and SVAMP with accuracies of 96.5% and 93.7%, respectively. Our code is publicly available at https://github.com/XuZhao0/Model-Selection-Reasoning.
BibTeX
@inproceedings{zhao2023automatic,
  author = {Xu Zhao and Yuxi Xie and Kenji Kawaguchi and Junxian He and Qizhe Xie},
  title = {{Automatic Model Selection with Large Language Models for Reasoning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.55},
}
Figure