Self-training Language Models for Arithmetic Reasoning

Marek Kadlcík, Michal StefánikEMNLP 2024 · DOI

Abstract

Recent advancements in large language models (LLMs) have shown significant potential in logical tasks, yet multi-step arithmetic reasoning remains a challenge. This paper explores self-training methodologies that leverage automated feedback to improve model performance without additional human-annotated data. We demonstrate that online self-training using preference optimization methods, such as Kahneman-Tversky Optimization (KTO), can achieve accuracy gains of up to 25.9% and maintain reasoning robustness where traditional supervised fine-tuning (SFT) fails.

Introduction

Arithmetic reasoning requires precise, multi-step logical chains. While scaling data often improves LLM performance, high-quality human-annotated reasoning data is prohibitively expensive. However, arithmetic possesses an inherent advantage: objective correctness. This study investigates whether models can "self-improve" by exploring various reasoning paths for known problems and learning from automated feedback. By treating correctness as a reward signal, we can iterate on existing datasets to extract deeper reasoning capabilities.

The Self-Training Framework

The researchers evaluated two primary architectures: offline and online self-training. In the offline setting, a model generates a static batch of predictions which are then used for fine-tuning. In contrast, the online setting involves a continuous loop where the model generates predictions, receives real-time feedback, and updates its parameters.

The self-training loop architecture comparing SFT and Preference variants.

Two training objectives were compared within these loops:

  1. Supervised Fine-Tuning (SFT): The model is trained to maximize the likelihood of generating reasoning paths that led to correct answers.
  2. Preference Optimization (PO): Methods like Direct Preference Optimization (DPO) and Kahneman-Tversky Optimization (KTO) train the model to distinguish between and prefer correct reasoning paths over incorrect ones.

Experimental Setup

The experiments utilized CALCFORMERS, 3-billion-parameter models integrated with calculator tools to isolate reasoning errors from simple arithmetic mistakes. The training prompts were derived from the Ape210K dataset (200,000+ problems). To evaluate generalization, the models were tested on five out-of-distribution datasets: GSM8K, AQuA-RAT, MAWPS, SVAMP, and ASDiv-A. Notably, no new problem prompts were introduced during the self-training phase.

Results and Analysis

The results indicate a clear superiority of online methods over offline ones. Online self-training yielded a 25.9% performance gain compared to 13.9% for the offline variant. This suggests that learning from the model's current distribution of errors is more effective than learning from a fixed historical snapshot.

Performance comparison of online self-training methods across arithmetic datasets.

A critical finding was the "SFT Collapse" in online settings. While SFT improved performance on the training distribution, it often led to degraded performance on complex tasks like GSM8K. Qualitative analysis revealed that SFT models learned to "shortcut" the reasoning process, omitting logical steps to reach the answer faster, which reduced their generalizability. Conversely, Preference Optimization (specifically KTO) acted as a stabilizer, forcing the model to maintain coherent reasoning chains. KTO improved GSM8K accuracy from 43.2% to 52.7% and Ape210K accuracy from 26.3% to 49.6%.

Conclusion

Self-training with automated feedback provides a powerful, data-efficient path for enhancing the reasoning capabilities of language models. While online training is computationally intensive, the resulting gains in accuracy and rationale quality—particularly when using preference optimization—outperform traditional supervised methods. Future work will explore extending these automated feedback loops to non-mathematical domains where correctness may be defined by logical consistency or coherence.

Summary updated

BibTeX
@inproceedings{kadlcik2024self,
  author = {Marek Kadlcík and Michal Stefánik},
  title = {{Self-training Language Models for Arithmetic Reasoning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.721},
}
Figure