Fine-Tuning Small Reasoning Models for Quantum Field Theory

Nathaniel S. Woodward, Zhiqi Gao, Yurii Kvasiuk, Kendrick M. Smith, Frederic Sala, Moritz MünchmeyerarXiv 2026 · arXiv

TL;DR

This paper presents the first academic fine-tuning study of small (7B-parameter) reasoning models specialized for theoretical physics, specifically Quantum Field Theory (QFT). By developing a novel pipeline that generates auto-verifiable physics problems, the researchers compare Reinforcement Learning (RL) and Supervised Fine-Tuning (SFT), revealing that domain-specific fine-tuning primarily improves factual grounding rather than core mathematical or logical reasoning.

What Problem Does This Paper Address?

While Large Language Models (LLMs) are increasingly applied to advanced scientific disciplines, we know surprisingly little about how domain-specific reasoning develops during training. Major industry labs rarely publish their training data, precise RL setups, or intermediate reasoning traces. Furthermore, advanced fields like theoretical physics suffer from a severe scarcity of open-source, verifiable training data.

To bridge this gap, the authors tackle the challenge of training and analyzing small reasoning models within academia. They focus on Quantum Field Theory (QFT)—a notoriously complex area of physics that serves as an ideal testing ground due to its structured pedagogical progression and analytically tractable calculations.

Methods

The researchers developed a robust data generation pipeline using frontier models (like Gemini and GPT-5) to create and filter physics problems. To make these complex problems verifiable without relying on brittle symbolic expression matching, they introduced an auto-verification method: models must implement their final analytic solution within a specified Python code skeleton.

Figure 2: A characteristic example of the Python code skeleton provided in a problem statement. The model converts its analytic solution into this function, enabling automated test case verification.

Using this data, they fine-tuned DeepSeek-R1-Distill-Qwen-7B using two distinct paradigms:

  1. Reinforcement Learning (RL): Utilizing Group Relative Policy Optimization (GRPO) with an accuracy-only binary reward function based on the automated test cases.
  2. Supervised Fine-Tuning (SFT): Distilling correct reasoning traces from larger open models (such as Qwen3-30B-A3B) using rejection sampling.

To understand how the models learned, the authors built a "Distill-then-Classify" pipeline that strips out verbose chain-of-thought (CoT) noise and categorizes reasoning errors into four types: Factual, Mathematical, Logical, and Executional.

Datasets

The authors curated and released several datasets categorized by operational difficulty:

  • Synthetic QFT Datasets: Consisting of Easy, Medium, and Hard tiers (~2,500 problems total) spanning undergraduate to post-graduate topics.
  • Human-Adapted Datasets: A evaluation set of over 800 problems adapted from standard QFT textbooks (e.g., Zee, Peskin & Schroeder), exercise books, MIT OpenCourseWare, and physics arXiv manuscripts.

The tasks within these datasets span multiple types, heavily favoring hidden-coefficient derivations.

Figure 3: Task distributions across the three synthetic training datasets. We find hidden-coefficient calcula- tions are the primary task select by generating models.

Key Insights and Ideas

  • Operational Complexity > Domain Difficulty: Surprisingly, a model's ability to solve a physics problem is driven by its operational complexity (the length of the derivation chain and number of intermediate steps) rather than its domain difficulty (how advanced the physics concept is). Advanced post-graduate problems with direct, single-step solutions were often solved more reliably than simpler undergraduate problems requiring long, multi-step calculations.
  • Finetuning Fixes Facts, Not Logic: Detailed error analysis revealed that both RL and SFT achieved their performance gains primarily by eliminating factual errors (misrecalling physics constants, sign conventions, or misreading the prompt). Core mathematical and algebraic manipulation errors remained the primary bottleneck post-training.
  • Compute Allocation Divergence: RL and SFT alter the model's internal reasoning behavior differently. RL training encourages the model to allocate more test-time compute (longer traces and more backtracking/self-correction) to problems it can successfully solve, while reducing futile loops on unsolvable ones. SFT, conversely, universally suppresses backtracking and inflates incorrect trace lengths due to imitating verbose teacher models.

Key Results

Fine-tuning yielded substantial performance jumps across both synthetic and out-of-distribution (OOD) human-adapted benchmarks.

Figure 1: Performance gains after fine-tuning DeepSeek-R1-Distill-Qwen-7B with RL on QFT Easy com- pared to SFT on Qwen3-30B-A3B correct reasoning traces on QFT Easy.
  • Synthetic Performance: On the Easy QFT validation set, the base model's accuracy rose from 40.2% to 54.2% via RL, and to 59.7% via SFT.
  • Zero-Shot Transfer: Training strictly on Easy problems induced a massive zero-shot transfer to Medium problems, pushing baseline accuracy from 26.2% up to 44.0% (RL) and 45.2% (SFT).
  • OOD Generalization: While SFT achieved higher peaks on in-distribution synthetic tasks, the RL-trained model generalized significantly better to human-authored physics problems, boosting arXiv benchmark performance from 16.6% to 24.6% (compared to SFT's 18.8%).

Limitations and Future Work

Despite these clear gains, the models hit a "Hard Reasoning Barrier"—the Hard QFT dataset remained largely unsolved, with the best fine-tuned model reaching only 2% accuracy.

Additionally, conducting RL fine-tuning remains highly compute-prohibitive for standard academic budgets; the RL pipeline for the Easy dataset alone required 160 hours on 4xH200 GPUs (640 GPU hours). The authors suggest that future academic work should move beyond simple outcome-based RL by integrating symbolic computation tools (like Mathematica or SymPy) directly into the training loop to alleviate the algebraic bottlenecks that small models face.

Summary updated

Abstract

Despite the growing application of Large Language Models (LLMs) to theoretical physics, there is little academic exploration into how domain-specific physics reasoning ability develops while training these models. To investigate this, we perform the first academic fine-tuning study of small (7B-parameter) reasoning models dedicated specifically to theoretical physics. Because open-source verifiable training data required to train such capabilities is scarce, we developed a robust data generation pipeline that can both create synthetic problems and make existing human-authored problems suitable for model training. Selecting Quantum Field Theory (QFT) as our primary domain, we generated over 2,500 synthetic problems alongside a curated collection of human-adapted problems sourced from arXiv and standard pedagogical resources. We conduct both Reinforcement Learning (RL) and Supervised Fine-Tuning (SFT) experiments, benchmarking performance gains as well as generalization to other physics domains. We perform an extensive analysis of model chains-of-though before and after fine-tuning, to understand how reasoning errors evolve during RL and SFT. Finally, we publicly release our data pipeline, verifiable QFT training data, and $\sim$200M tokens of QFT reasoning traces.
BibTeX
@misc{woodward2026fine,
  author = {Nathaniel S. Woodward and Zhiqi Gao and Yurii Kvasiuk and Kendrick M. Smith and Frederic Sala and Moritz Münchmeyer},
  title = {{Fine-Tuning Small Reasoning Models for Quantum Field Theory}},
  year = {2026},
  eprint = {2604.18936},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.18936},
}

Loading related content...

Figure