TL;DR
This paper investigates how different domain specialization techniques for Large Language Models (LLMs) – specifically fine-tuning (SFT) and in-context learning (ICL) – impact the characteristics of generated text in the highly specialized domain of accounting audit procedures. The study reveals that while both methods improve performance, they lead to distinct differences in specificity and relevance, with hybrid approaches showing additive benefits.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly being adapted for specialized domains through techniques like fine-tuning (SFT) and in-context learning (ICL). While prior research has compared their task-specific capabilities, there's a gap in understanding how these specialization methods influence the qualitative characteristics of the generated text itself. This is particularly important in domains like accounting audit procedures, which require expert knowledge and precise language. The authors aim to address this by analyzing the generated audit procedures from multiple perspectives, going beyond simple performance metrics to understand the nuances of domain specialization.
Methods
The researchers employed a multi-perspective evaluation framework using an "LLM-as-a-judge" approach. They compared three main domain specialization strategies:
- Supervised Fine-Tuning (SFT): This involved adapting pre-trained LLMs to the accounting audit domain. They explored two variants: SFT-IT (fine-tuning on an instruction-tuned model) and SFT-CV (fine-tuning with a "chat vector" to incorporate conversational behavior). They used QLoRA for efficient fine-tuning.
- In-Context Learning (ICL): This method involves providing the LLM with a few demonstration examples within the prompt during inference. The study explored different strategies for selecting these few-shot examples, including nearest neighbor and diversity-based approaches.
- Hybrid Approaches (SFT-FS): This combines supervised fine-tuning with few-shot examples, inspired by Retrieval Augmented Fine-Tuning (RAFT).
The generated audit procedures were evaluated using four key metrics: accuracy, comprehensiveness, specificity, and relevance, derived from the requirements of accounting audits.
Datasets
The primary dataset used in this study is the Accounting Audit Procedures Generation (AAPG) task dataset. This dataset was extracted from audit reports of securities filings between March 31, 2021, and March 31, 2024. It contains Key Audit Matters (KAMs) and corresponding auditors' responses.
For generalizability testing, experiments were also conducted on other Long-Form Question Answering (LFQA) datasets:
- MilkQA: Consumer questions and expert answers from the dairy sector.
- cMedQA2: Questions and answers from a Chinese online health consultation website.
- ELI5-Category: Question-answer pairs from Reddit's "r/explainlikeimfive" subreddit.
Key Insights and Ideas
- Distinct Generative Signatures: SFT and ICL, while both improving performance, imprint different characteristics onto the generated text. SFT excels in relevance, while ICL is stronger in specificity.
- Additive Effects of Hybridization: Combining SFT and ICL (as in SFT-FS) often leads to additive improvements, meaning the combined approach performs better than either method alone, rather than just a simple union of their capabilities.
- ICL as Knowledge Injection: The study suggests that ICL in this context primarily functions by injecting relevant contextual knowledge, rather than just pattern recognition of input-output pairs.
- SFT-CV's Robustness: The SFT-CV variant showed more consistent improvements and less performance degradation compared to SFT-IT, especially in cross-dataset evaluations.
- Domain Specialization Matters: The AAPG task, being highly specialized, showed significant potential for domain specialization, with ground truth evaluations consistently outperforming vanilla LLMs.
Key Results
- Overall Improvement: All domain specialization approaches (SFT, ICL, and hybrid) showed improvements over vanilla LLMs across all four metrics (accuracy, comprehensiveness, specificity, relevance) on the AAPG task.
- Specificity vs. Relevance: SFT-based methods consistently underperformed ICL in specificity, while SFT-based methods consistently improved over ICL in relevance.
- Hybrid Performance: Hybrid approaches generally outperformed individual methods. SFT-FS, in particular, demonstrated greater improvements in relevance compared to other methods.
- Model Differences: The effectiveness of ICL's few-shot selection strategies varied across models. For Qwen2, 20-nearest examples were most effective, while Swallow and Llama-3.1 performed best with fewer, more diverse examples.
- Cross-Dataset Generalizability: While AAPG showed clear benefits from specialization, results on MilkQA and cMedQA2 were mixed, suggesting that dataset quality and domain specificity play a crucial role in observing these improvements. For instance, on cMedQA2, some specialized methods showed performance degradation.
Limitations and Future Work
- Narrow Domain Focus: The study was limited to audit procedure generation, a highly specialized task. Replicating these findings on other LFQA datasets with higher-quality ground truth may present challenges.
- Model Size: Experiments were conducted on LLMs around 7-8B parameters. It's unclear if similar results would hold for smaller or larger models, as model size can influence specialization effectiveness.
- PEFT vs. Full Parameter Models: The use of PEFT (QLoRA) for fine-tuning raises questions about whether full-parameter fine-tuning or PEFT without quantization would yield similar outcomes.
- ICL Ablation Constraints: Computational resources limited the range of 'k' (number of examples) explored in the ICL ablation study.
- LLM-as-a-Judge Reliability: While sensitivity analysis was performed, the LLM-as-a-judge approach might still have gaps between its evaluation and human perception.
Future work could explore these limitations, particularly by investigating different model sizes and full-parameter fine-tuning. Further research on higher-quality, diverse LFQA datasets is also recommended.
Summary updated