TL;DR
The Chain-of-Thought (CoT) Hub is an open-source evaluation suite designed to measure the multi-step reasoning capabilities of Large Language Models (LLMs) using few-shot chain-of-thought prompting. The benchmark reveals a stark performance gap between leading commercial models (like GPT-4 and Claude) and open-source alternatives, while demonstrating that model scale and Reinforcement Learning from Human Feedback (RLHF) are the primary drivers of advanced reasoning performance.
What Problem Does This Paper Address?
Evaluating modern LLMs is notoriously difficult. When restricted to casual chitchat, smaller or open-source models (like Alpaca or Vicuna) can superficially seem just as capable as proprietary giants like GPT-3.5. However, this conversational veneer thins out when models face complex, multi-step tasks.
As the AI ecosystem shifts toward using LLMs as foundational computational engines—powering orchestration frameworks like LangChain—the ability to compose linguistic and logical operations becomes paramount. The authors argue that complex reasoning is the true differentiator between weak and strong LLMs. To track progress accurately, the community needs a centralized, challenging benchmark suite focused specifically on these multi-step capabilities rather than simple answer-only or dialogue-preference metrics.
Methods
The authors introduce the Chain-of-Thought Hub, a continuous evaluation framework characterized by three core methodological choices:
- Few-Shot Chain-of-Thought Prompting: Unlike concurrent evaluation suites (such as HELM) that rely primarily on answer-only prompting, CoT Hub evaluates models by prompting them to generate intermediate reasoning steps.
- Few-Shot over Zero-Shot: The authors intentionally use few-shot exemplars because this capability exists in both raw pretrained base models and instruction-tuned checkpoints, preventing the underestimation of base models.
- Final Answer Accuracy as a Proxy: When scoring, the framework evaluates only the correctness of the final answer (e.g., a final number or multiple-choice option) rather than manually grading the intermediate steps. Empirically, the authors note that if the intermediate logic is flawed, the model rarely stumbles into the correct final answer.
Datasets
CoT Hub compiles six challenging, widely recognized benchmarks comprising over 100 subtasks:
- GSM8k: 8k grade-school math word problems testing arithmetic and linguistic multi-step reasoning.
- MATH: 12k extremely difficult high school competition-level math problems written in LaTeX.
- MMLU: 15k problems spanning 57 subjects across humanities and STEM to test high-school and college-level knowledge.
- BigBench Hard (BBH): A 6.5k-problem subset of BigBench specifically isolated because it requires chain-of-thought prompting to solve.
- HumanEval: 164 handwritten Python programming challenges evaluating code generation.
- C-Eval: A comprehensive Chinese evaluation suite containing 13k multiple-choice questions across 52 disciplines.
Key Insights and Ideas
- Reasoning Scales Log-Linearly: Model performance on reasoning tasks scales predictably with parameter count. When plotting accuracy against the log of model scale, the trend line is remarkably linear.
- RLHF is the "Moat": Except for PaLM-2, the top-performing models on the leaderboard all utilize RLHF. This strongly implies that standard supervised instruction finetuning (SIFT) is insufficient for maximizing an LLM's latent reasoning potential.
- The Latent Power of LLaMA-65B: LLaMA-65B (an open-source base model) performs remarkably close to OpenAI’s
code-davinci-002(the base model for the GPT-3.5 series). This is a massive "aha" moment for the open-source community: it suggests that if developers apply proper RLHF alignment to LLaMA-65B, it could realistically perform on par with ChatGPT-3.5.
Key Results
The evaluation ranks 19 major language models.
- The Leaderboard Hierarchy: As of May 2023, GPT-4 reigns supreme, achieving a dominant 92.0% on GSM8k and 86.4% on MMLU. Only Anthropic’s Claude-v1.3 (81.8% GSM8k) and Google’s PaLM-2 (80.7% GSM8k) are competitive.
- Open-Source Lag: Open-source models clearly lag behind commercial APIs. The strongest open-source model, LLaMA-65B, scores 50.9% on GSM8k and 63.4% on MMLU, significantly outperforming smaller instruction-tuned models like Flan-T5 but remaining far behind GPT-4.
The complete breakdown of the evaluation suite results is detailed below:
Limitations and Future Work
A primary limitation is the lack of public access to certain proprietary models (like the full PaLM-2 family), forcing the authors to rely on figures reported in technical papers rather than direct execution.
Moving forward, the authors plan to continuously update the CoT Hub by:
- Expanding the suite to include commonsense reasoning, formal math theorem proving, and tool/API-calling benchmarks.
- Integrating newer open-source instruction-tuned variants (like Vicuna) and alternative commercial APIs.
- Exploring hybrid methods to tackle the grueling MATH benchmark, such as prompting LLMs to interactively call external symbolic calculators like Wolfram Alpha.
Summary updated