VCR: A "Cone of Experience" Driven Synthetic Data Generation Framework for Mathematical Reasoning

Sannyuya Liu, Jintian Feng, Xiaoxuan Shen, Shengyingjie Liu, Qian Wan, Jianwen SunAAAI 2025 · OpenReview

Abstract

Large language models (LLMs) have shown excellent performance in natural language processing but struggle with mathematical reasoning. As the training mode gradually solidifies, researchers propose a data-centric concept of artificial intelligence, emphasizing the development of higher-quality data to empower LLMs. Existing studies construct synthetic data for mathematical reasoning by expanding public datasets, thereby performing supervised fine-tuning of LLMs. However, these methods mostly focus on quantity while neglecting quality. The challenging samples fail to receive adequate consideration during the data synthesis process, resulting in high construction costs, low-quality density, and serious data homogenization. This paper proposes a multi-agent environment called Virtual ClassRoom (VCR), which leverages various agents driven by LLM to construct high-quality diversified synthetic data. Inspired by the "Cone of Experience" educational theory, VCR introduces three experience levels (direct, iconic, and symbolic) into the data synthesis process by analogy with human learning. A user-friendly instruction set and role-playing system are carefully designed, enabling VCR to autonomously plan the scale of synthetic data. This system covers various educational scenarios, including lecture, discussion, problem design, and problem-solving. The Adaboost idea embodied in the global iterative process further promotes steady performance improvement. Extensive experiments show that the synthetic data generated by VCR possess higher quality density and generalization capability, which can give LLMs superior mathematical reasoning performance with the same scale.

Introduction

While Large Language Models (LLMs) excel at general language tasks, they frequently struggle with complex mathematical reasoning. To address this, researchers typically employ Supervised Fine-Tuning (SFT) using synthetic datasets generated by more powerful models like GPT-4. However, current data generation methods, such as MetaMathQA or XwinMathQA, often rely on a "brute-force" approach—blindly rewriting or augmenting massive amounts of queries and responses.

This strategy often ignores data quality and density, wasting computational resources on generating variations of simple questions the model already understands while failing to provide targeted practice for challenging problems. The result is high construction costs and severe data homogenization. To overcome these limitations, we introduce the Virtual ClassRoom (VCR), a smarter, more targeted data generation pipeline that mimics human learning processes.

The Virtual ClassRoom (VCR) Framework

VCR is a multi-agent environment powered by AutoGen that assigns LLM agents to specific roles—Teacher, Teaching Assistant (TA), Student, and Noter—to simulate a classroom setting. The workflow is structured around Edgar Dale’s "Cone of Experience" theory, divided into three distinct stages:

The Virtual ClassRoom (VCR) framework and the Cone of Experience.
  1. Learning by Doing (Direct Experience): Before generating data, the target LLM undergoes a diagnostic "test" on a base dataset. VCR calculates error rates for each question and, inspired by the AdaBoost algorithm, allocates a higher synthetic data generation budget to the questions the LLM failed. This ensures the new data specifically targets the model's weaknesses.
  2. Learning through Observation (Iconic Experience): This stage simulates classroom interactions. The Teacher agent provides detailed lectures (augmenting answers), while Student agents engage in discussions. This process generates multiple perspectives and a "problem design mind," capturing the underlying logic and intent of the queries.
  3. Learning through Abstractions (Symbolic Experience): The TA agent reviews key information from previous steps and designs brand-new, similar questions tailored to the LLM's identified weaknesses, facilitating reflection and higher-level abstraction.

This process is governed by a Global Iterative Process, where VCR re-evaluates the LLM after each training epoch, dynamically adjusting the data generation focus as the model improves.

Experimental Results

We evaluated VCR using base training data from GSM8K and MATH, testing across multiple benchmarks including SVAMP, ASDiv, and GSM-Distractor.

Performance comparison of VCR against state-of-the-art methods on GSM8K and MATH benchmarks.

Massive Efficiency and Performance Gains

Using a strict budget of only 60,000 synthetic samples, VCR outperformed baselines across multiple base models, including LLaMA-2 (7B/13B/70B), Mistral-7B, and LLaMA-3-8B. Notably, VCR achieved a +5.6% to +10.3% boost on GSM8K and a +5.6% to +7.8% boost on MATH compared to MetaMathQA, which uses 395,000 samples. Furthermore, VCR matched or exceeded the performance of SkyworkMathQA (SMQ), which utilizes a massive 2.5 million samples. For LLaMA-2-7B, VCR outperformed SMQ by +1.5% on GSM8K and +3.0% on MATH using a fraction of the data.

The Power of "Indirect" Data

A key insight from our experiments is the value of the "problem design mind"—the simulated students' thought process regarding how and why a math question is designed. Including this meta-process in the training data significantly boosted reasoning stability and performance. Structuring LLM data generation like a human classroom (lecture → discussion → reflection) yields higher quality, more diverse data than simple prompt rewriting.

Conclusion

The VCR framework demonstrates that analogizing human learning experiences provides a powerful new paradigm for SFT data construction. By focusing on quality density and targeted generation rather than sheer quantity, VCR unlocks superior mathematical reasoning capabilities in LLMs. Future work will explore other types of "indirect data" to further enhance model performance.

Summary updated

Abstract

Large language models (LLMs) have shown excellent performance in natural language processing but struggle with mathematical reasoning. As the training mode gradually solidifies, researchers propose a data-centric concept of artificial intelligence, emphasizing the development of higher-quality data to empower LLMs. Existing studies construct synthetic data for mathematical reasoning by expanding public datasets, thereby performing supervised fine-tuning of LLMs. However, these methods mostly focus on quantity while neglecting quality. The challenging samples fail to receive adequate consideration during data synthesis process, resulting in high construction costs, low-quality density, and serious data homogenization. This paper proposes a multi-agent environment called Virtual ClassRoom (VCR), which leverages various agents driven by LLM to construct high-quality diversified synthetic data. Inspired by the "Cone of Experience" educational theory, VCR introduces three experience levels (direct, iconic, and symbolic) into data synthesis process by analogy with human learning. A user-friendly instruction set and role-playing system are carefully designed, enabling VCR to autonomously plan the scale of synthetic data. This system covers various educational scenarios, including lecture, discussion, problem design and problem-solving. The Adaboost idea embodied in the global iterative process further promotes steady performance improvement. Extensive experiments show that the synthetic data generated by VCR possess higher quality density and generalization capability, which can give LLMs superior mathematical reasoning performance with the same scale.
BibTeX
@inproceedings{liu2025vcr,
  author = {Sannyuya Liu and Jintian Feng and Xiaoxuan Shen and Shengyingjie Liu and Qian Wan and Jianwen Sun},
  title = {{VCR: A "Cone of Experience" Driven Synthetic Data Generation Framework for Mathematical Reasoning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=9cUdYP7dTp},
}
Figure