Exploring the Vertical-Domain Reasoning Capabilities of Large Language Models

Jie Zhou, Xin Chen, Jie Zhang, Zhe LiarXiv 2025 · DOI · arXiv

Abstract

This research investigates the "accounting reasoning" capabilities of Large Language Models (LLMs), defined as a unique blend of multi-step mathematics and strict regulatory logic. By introducing a specialized benchmark, the authors reveal that while top-tier models like GPT-4 show promise, they still struggle with error propagation and complex rule application. The findings suggest that current LLMs are not yet ready for autonomous professional use in highly regulated vertical domains.

Introduction

While LLMs have mastered general conversation and basic coding, their performance in "vertical domains"—specialized fields like law, medicine, or accounting—remains inconsistent. Accounting presents a specific challenge: it requires more than just arithmetic. Models must adhere to rigid standards (such as GAAP or IFRS), maintain numerical consistency across multiple financial statements, and execute long chains of "if-then" logic.

Existing benchmarks are often too general or too shallow. There is a significant gap in understanding how LLMs handle the "reasoning chain" required for complex, multi-step accounting problems where a single initial error cascades through the entire result.

Methodology

The study formally defines Accounting Reasoning as a composite capability involving mathematical computation, logical deduction, and domain-specific constraint satisfaction. To evaluate this, the researchers employed:

  • Few-shot Chain-of-Thought (CoT) Prompting: Encouraging models to "think out loud" by providing examples of step-by-step reasoning.
  • Comparative Analysis: Testing OpenAI’s GPT-4 against the GLM series (GLM-6B, 130B, and GLM-4) to compare general-purpose models with those potentially exposed to more region-specific financial data.
  • Error Taxonomy: Categorizing failures into types such as "logic misuse," "incomplete knowledge," and "calculation errors" to pinpoint specific failure modes.

Evaluation Framework

The evaluation utilized three primary components:

  1. Multi-Calculation Benchmark: 586 problems from MR-GSM8K requiring at least three computational steps to test long-chain accuracy.
  2. Accounting-Reasoning-Benchmark: A new dataset covering accounting principles, financial statement analysis, cost accounting, and auditing.
  3. Contextual Benchmarks: References to Chinese benchmarks like CLUE and professional CPA exam questions for baseline knowledge.

Key Insights

The Cascade Effect and Error Propagation

The primary hurdle in accounting reasoning is not the arithmetic itself, but the propagation of errors. A minor mistake in an initial asset valuation invalidates every subsequent calculation, such as depreciation, tax, and net income.

Principles vs. Procedures

"Principle-level" misunderstandings (e.g., misclassifying an asset) are significantly more damaging than "procedural" errors (e.g., ledger typos). These conceptual failures fundamentally derail the logic of the entire task.

Domain-Specific Advantages

GLM-4 slightly outperformed GPT-4 in specific accounting knowledge tasks. This suggests that models trained with a stronger focus on regional regulatory environments (like Chinese accounting standards) may possess a "home-field advantage" in vertical domains.

Results and Analysis

Numerical Stability

On the Multi-Calculation Benchmark, GPT-4 demonstrated superior performance in general multi-step math.

Multi-Calculation Benchmark performance across models showing GPT-4's lead in numerical stability.

As shown in Figure 1, GPT-4 achieved 91.45% accuracy, while the smaller GLM-6B struggled at 19.43%.

Accounting Accuracy and Difficulty Scaling

Performance dropped significantly when domain-specific logic was introduced. On the specialized Accounting Reasoning Benchmark, GLM-4 achieved 21.78% and GPT-4 achieved 16.58%.

Comparison of GPT-4 and GLM-4 accuracy across difficulty levels in the accounting domain.

As tasks became more complex, the performance gap shifted. For "Hard" level questions, GPT-4's accuracy fell to 5.11%, while GLM-4 maintained a slightly better 13.87% (Figure 2). Analysis showed that over 50% of errors resulted from misunderstanding accounting principles and insufficient conceptual coverage.

Conclusion and Future Work

Current LLMs face significant challenges in vertical-domain reasoning. The study highlights that moving from "Zero-shot" to "Few-shot CoT" improves accuracy by approximately 50%, yet professional-grade reliability remains elusive.

Future research should focus on:

  • Domain-Specific Fine-Tuning: Moving beyond general-purpose training.
  • Knowledge Graph Integration: Providing LLMs with an explicit "rulebook" to guide the reasoning process.
  • Real-World Data: Testing against the messy, unstructured nature of actual corporate financial data.

Summary updated

BibTeX
@misc{zhou2025exploring,
  author = {Jie Zhou and Xin Chen and Jie Zhang and Zhe Li},
  title = {{Exploring the Vertical-Domain Reasoning Capabilities of Large Language Models}},
  year = {2025},
  eprint = {2512.22443},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.22443},
}
Figure