Towards Reasoning Ability of Small Language Models

Gaurav Srivastava, Shuxiang Cao, Xuan WangarXiv 2025 · DOI · arXiv

Introduction

The prevailing narrative in large language model (LLM) development has long suggested that complex multi-step reasoning is an "emergent property" exclusive to models exceeding 100 billion parameters. This assumption has driven the industry toward massive, resource-intensive architectures. However, the demand for privacy, low latency, and cost-efficiency has spurred interest in Small Language Models (SLMs). This paper introduces THINKSLM, a systematic framework to evaluate whether SLMs (<30B parameters) truly possess reasoning capabilities or merely excel at pattern memorization.

The THINKSLM Framework

To rigorously assess SLMs, the researchers developed THINKSLM, evaluating 72 models across six dimensions: capacity, training recipes, prompting, compression, task domains, and robustness.

Methodological Innovations

  • LLM-as-a-Judge: The study validated automated evaluation by comparing GPT-4-Turbo and GPT-4o against human judgment, finding a 98% alignment in generative reasoning tasks.
  • Algorithmic Stress Testing: To avoid data contamination, a custom SORTING task was introduced. By varying list lengths and numerical types (e.g., negative numbers), the task tests pure symbolic manipulation rather than memorized sequences.
  • Comprehensive Benchmarking: The framework utilizes 17 benchmarks, including GSM8K (math), ARC (science), and GSM-Plus (adversarial robustness).

Drivers of Reasoning in SLMs

The analysis reveals that reasoning ability is not solely a function of scale but is heavily influenced by data quality and training alignment.

  • Data Quality vs. Scale: The Qwen2.5-7B model significantly outperformed similarly sized counterparts like Mistral-7B by over 30% on mathematical tasks. This indicates that "reasoning-rich" pre-training data is a primary driver of performance.
  • Internalized Reasoning: For models released after 2024, explicit Chain-of-Thought (CoT) prompting yielded only marginal gains (~2%). This suggests that modern SLMs have internalized step-by-step reasoning during the alignment phase.

Impact of Model Compression

A critical aspect of deploying SLMs is efficiency. The study compared quantization and pruning to understand their impact on the model's logical "brain."

Quantization performance across GSM8K, ARC/CQA, and Sorting tasks, showing that 4-bit and 8-bit GPTQ maintain high accuracy compared to non-quantized models.

The researchers found that quantization is essentially a "free lunch." As shown in the figure above, 4-bit quantization—which reduces memory usage by up to 80%—has a negligible impact on reasoning accuracy. In contrast, structured pruning is a "reasoning killer." Removing weights to create sparsity significantly disrupts internal logic, with a pruned Llama-8B model experiencing a 32% drop in math accuracy.

Benchmarking Results and Robustness

The evaluation demonstrates that top-tier SLMs can rival proprietary giants in specific reasoning domains.

Overall performance landscape (a), sorting task performance (b), and the effect of prompting strategies (c) across a wide range of model parameters.

Key Findings

  • Competitive Parity: Models like Qwen2.5-32B perform comparably to GPT-4-Turbo on intermediate reasoning tasks.
  • Robustness Scaling: While smaller models can achieve high accuracy, larger models remain more robust. Adversarial perturbations (GSM-Plus) caused performance drops of 11–17% across all models, though relative rankings remained stable.
  • The Algorithmic Gap: Even the most capable models, such as Llama-3-70B, showed performance degradation as sorting tasks increased in complexity, suggesting a continued reliance on shallow heuristics for algorithmic logic.

Conclusion

The THINKSLM benchmark demonstrates that with high-quality data and strategic alignment, SLMs can achieve reasoning capabilities previously thought to be reserved for massive models. While quantization offers a viable path for efficient deployment without sacrificing logic, challenges remain in instruction following and robust algorithmic reasoning. Future work will focus on closing these gaps to enable more capable local AI.

Summary updated

BibTeX
@misc{srivastava2025towards,
  author = {Gaurav Srivastava and Shuxiang Cao and Xuan Wang},
  title = {{Towards Reasoning Ability of Small Language Models}},
  year = {2025},
  eprint = {2502.11569},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.11569},
}
Figure