Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance

Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, Tushar Khot

TL;DR

The Chain-of-Thought (CoT) Hub is an open-source evaluation suite designed to measure the multi-step reasoning capabilities of Large Language Models (LLMs) using few-shot chain-of-thought prompting. The benchmark reveals a stark performance gap between leading commercial models (like GPT-4 and Claude) and open-source alternatives, while demonstrating that model scale and Reinforcement Learning from Human Feedback (RLHF) are the primary drivers of advanced reasoning performance.

What Problem Does This Paper Address?

Evaluating modern LLMs is notoriously difficult. When restricted to casual chitchat, smaller or open-source models (like Alpaca or Vicuna) can superficially seem just as capable as proprietary giants like GPT-3.5. However, this conversational veneer thins out when models face complex, multi-step tasks.

As the AI ecosystem shifts toward using LLMs as foundational computational engines—powering orchestration frameworks like LangChain—the ability to compose linguistic and logical operations becomes paramount. The authors argue that complex reasoning is the true differentiator between weak and strong LLMs. To track progress accurately, the community needs a centralized, challenging benchmark suite focused specifically on these multi-step capabilities rather than simple answer-only or dialogue-preference metrics.

Methods

The authors introduce the Chain-of-Thought Hub, a continuous evaluation framework characterized by three core methodological choices:

  1. Few-Shot Chain-of-Thought Prompting: Unlike concurrent evaluation suites (such as HELM) that rely primarily on answer-only prompting, CoT Hub evaluates models by prompting them to generate intermediate reasoning steps.
  2. Few-Shot over Zero-Shot: The authors intentionally use few-shot exemplars because this capability exists in both raw pretrained base models and instruction-tuned checkpoints, preventing the underestimation of base models.
  3. Final Answer Accuracy as a Proxy: When scoring, the framework evaluates only the correctness of the final answer (e.g., a final number or multiple-choice option) rather than manually grading the intermediate steps. Empirically, the authors note that if the intermediate logic is flawed, the model rarely stumbles into the correct final answer.

Datasets

CoT Hub compiles six challenging, widely recognized benchmarks comprising over 100 subtasks:

  • GSM8k: 8k grade-school math word problems testing arithmetic and linguistic multi-step reasoning.
  • MATH: 12k extremely difficult high school competition-level math problems written in LaTeX.
  • MMLU: 15k problems spanning 57 subjects across humanities and STEM to test high-school and college-level knowledge.
  • BigBench Hard (BBH): A 6.5k-problem subset of BigBench specifically isolated because it requires chain-of-thought prompting to solve.
  • HumanEval: 164 handwritten Python programming challenges evaluating code generation.
  • C-Eval: A comprehensive Chinese evaluation suite containing 13k multiple-choice questions across 52 disciplines.

Key Insights and Ideas

  • Reasoning Scales Log-Linearly: Model performance on reasoning tasks scales predictably with parameter count. When plotting accuracy against the log of model scale, the trend line is remarkably linear.
  • RLHF is the "Moat": Except for PaLM-2, the top-performing models on the leaderboard all utilize RLHF. This strongly implies that standard supervised instruction finetuning (SIFT) is insufficient for maximizing an LLM's latent reasoning potential.
  • The Latent Power of LLaMA-65B: LLaMA-65B (an open-source base model) performs remarkably close to OpenAI’s code-davinci-002 (the base model for the GPT-3.5 series). This is a massive "aha" moment for the open-source community: it suggests that if developers apply proper RLHF alignment to LLaMA-65B, it could realistically perform on par with ChatGPT-3.5.

Key Results

The evaluation ranks 19 major language models.

Figure 1. X-axis means the log of the model scale measured in billion parameters. We observe that model performance is generally correlated with scale, approximately showing a log-linear trend. Models without disclosing their scale generally perform better than models disclosing scale information. Our observations also indicate that the open-source community may still needs to explore/ figure out “the moat” about the scaling and RLHF for further improvements.
  • The Leaderboard Hierarchy: As of May 2023, GPT-4 reigns supreme, achieving a dominant 92.0% on GSM8k and 86.4% on MMLU. Only Anthropic’s Claude-v1.3 (81.8% GSM8k) and Google’s PaLM-2 (80.7% GSM8k) are competitive.
  • Open-Source Lag: Open-source models clearly lag behind commercial APIs. The strongest open-source model, LLaMA-65B, scores 50.9% on GSM8k and 63.4% on MMLU, significantly outperforming smaller instruction-tuned models like Flan-T5 but remaining far behind GPT-4.

The complete breakdown of the evaluation suite results is detailed below:

Table 1. Overall model performance on Chain-of-Thought Hub. Numbers with an asterisk* are from our test scripts. For model types, base means the model checkpoint after pretraining, SIFT means supervised instruction finetuning. Others are from their corresponding papers. We observe: (1) there exist a gap between leading LLMs (GPT, Claude and PaLM) and open-source (LLaMA and FlanT5); (2) most leading LLMs are after RLHF, indicating the opportunity of improving open-sourced models using this technique; (3). model performance is generally correlated with model scale, indicating further opportunities in scaling, especially for open-source models. We further highlight that among open-sourced models, LLaMA 65B performs close to code-davinci-002, the base model of ChatGPT. This suggests that if RLHF is done right on LLaMA 65B, it may become close to ChatGPT.

Limitations and Future Work

A primary limitation is the lack of public access to certain proprietary models (like the full PaLM-2 family), forcing the authors to rely on figures reported in technical papers rather than direct execution.

Moving forward, the authors plan to continuously update the CoT Hub by:

  1. Expanding the suite to include commonsense reasoning, formal math theorem proving, and tool/API-calling benchmarks.
  2. Integrating newer open-source instruction-tuned variants (like Vicuna) and alternative commercial APIs.
  3. Exploring hybrid methods to tackle the grueling MATH benchmark, such as prompting LLMs to interactively call external symbolic calculators like Wolfram Alpha.

Summary updated

Abstract

As large language models (LLMs) are continuously being developed, their evaluation becomes increasingly important yet challenging. This work proposes Chain-of-Thought Hub, an open-source evaluation suite on the multi-step reasoning capabilities of large language models. We are interested in this setting for two reasons: (1) from the behavior of GPT and PaLM model family, we observe that complex reasoning is likely to be a key differentiator between weaker and stronger LLMs; (2) we envisage large language models to become the next-generation computational platform and foster an ecosystem of LLM-based new applications, this naturally requires the foundation models to perform complex tasks that often involve the composition of linguistic and logical operations. Our approach is to compile a suite of challenging reasoning benchmarks to track the progress of LLMs. Our current results show that: (1) model scale clearly correlates with reasoning capabilities; (2) As of May 2023, Claude-v1.3 and PaLM-2 are the only two models that are comparable with GPT-4, while open-sourced models still lag behind; (3) LLaMA-65B performs closely to code-davinci-002, indicating that with successful further development such as reinforcement learning from human feedback (RLHF), it has great potential to be close to GPT-3.5-Turbo. Our results also suggest that for the open-source efforts to catch up, the community may focus more on building better base models and exploring RLHF.
BibTeX
@inproceedings{fu2023chain,
  author = {Yao Fu and Litu Ou and Mingyu Chen and Yuhao Wan and Hao Peng and Tushar Khot},
  title = {{Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance}},
  booktitle = {DeployableGenerativeAI @ ICML},
  year = {2023},
  url = {https://openreview.net/forum?id=iHwy0EcGB8},
}
Figure