Introduction
While Large Language Models (LLMs) like BloombergGPT and FinGPT have made significant strides in the financial sector, their applications have largely been confined to broad tasks such as sentiment analysis, entity recognition, and general financial Q&A. In the specialized domain of credit loan business—particularly within China's "Inclusive Finance" and "Rural Finance" sectors—there remains a significant gap. Day-to-day operations require highly personalized customer service, including understanding complex loan requirements, comparing similar products, and matching loans to specific business metrics like operating time or annual revenue.
Existing AI solutions often focus on backend tasks like fraud detection or credit rating. CreditLLM addresses the lack of specialized, high-quality training data for front-end credit product customization and customer counseling by demonstrating that LLMs can be effectively fine-tuned for complex business workflows using a structured, small-scale seed dataset.
Methodology
To overcome the scarcity of proprietary banking records, the researchers developed a pipeline to transform a small amount of seed data into a robust instruction-tuning set.
Hierarchical Data Structuring and Prompt Generation
The team manually collected 238 real-world credit products from nine domestic Chinese banks, extracting structured key-value pairs such as loan limits, interest rates, and applicable objects. To simulate realistic customer-agent interactions, they developed a 10-category framework for customer communication.
As shown in the framework above, tasks were divided into specific and non-specific product categories, covering everything from product comparison to professional knowledge Q&A. Using manual rules based on this framework, the team generated 52,751 simulated dialogue prompts from the initial 238 products.
Training Strategy
The training process followed a two-stage approach:
- Concept Alignment (Continual Pre-Training): A Chinese financial foundation model (CFGPT) was pre-trained on a mix of general financial corpora, credit textbooks, and a subset of the generated credit data to align the model with domain-specific terminology.
- Instruction Tuning (Supervised Fine-Tuning): Using QLoRA, the model was fine-tuned on a mixed dataset containing general Chinese dialogues and the bulk of the generated credit prompts. This mixing was crucial to prevent the model from losing its general reasoning flexibility.
Finally, during inference, CreditLLM utilizes Retrieval-Augmented Generation (RAG) to pull the most relevant product details into the prompt context, ensuring responses are grounded in actual, up-to-date product specifications.
Dataset Composition
The resulting dataset, CreditData, consists of over 52,000 instruction-following prompts. The distribution of these tasks highlights the focus on complex reasoning such as product comparison and recommendation.
In addition to the generated prompts, the training pipeline incorporated existing datasets like FinCorpus, CFData, and COIG-CQIA to maintain broader financial comprehension and basic conversational abilities.
Key Results and Insights
CreditLLM was evaluated against baseline models using F1 scores for fixed-option tasks and cosine similarity for open-ended responses. The results demonstrated superior domain performance:
- Specialized Task Mastery: CreditLLM significantly outperformed the base CFGPT model. In Keyword-based Recommendation, CreditLLM achieved a score of 0.726 (vs. 0.474 for the base model), and in Attribute Matching, it scored 0.539 (vs. 0.429).
- The Power of Micro-Data: The study proves that massive datasets are not always a prerequisite for teaching an LLM a highly specific business skill. Rigorous structuring of a small amount of seed data (238 products) can impart complex reasoning skills like multi-condition product matching.
Limitations and Future Outlook
A primary challenge identified was catastrophic forgetting. While CreditLLM excels in its niche, its performance on general financial tasks (like Entity Recognition) degraded. Future work will focus on using advanced chat-based LLMs for data generation, exploring methodologies to alleviate forgetting, and implementing memory-editing techniques to manage outdated content. CreditLLM serves as a blueprint for building specialized financial assistants where data is scarce but business logic is complex.