TL;DR
This paper proposes a novel hybrid approach to classify companies into industry sectors using financial statements. By combining numerical data with textual information and leveraging advanced language models, the authors achieve high classification accuracy and offer explainable predictions, paving the way for more efficient and insightful auditing processes.
What Problem Does This Paper Address?
Auditing is a labor-intensive process that requires auditors to meticulously examine financial reports. A crucial step in this process is understanding a company's industry sector, which helps auditors prioritize areas of focus and identify potential risks. While existing industry classification systems like SIC and NAICS exist, and some machine learning approaches have been applied to financial data, there's a gap in effectively leveraging the rich, often textual, information within financial statements for accurate and explainable company classification. This paper aims to bridge this gap by exploring how to combine numerical and textual financial data using advanced AI techniques, specifically large language models (LLMs), to automate and improve this classification task.
Methods
The authors explore several methods for representing financial statements and classifying companies:
- Data Representations:
- Sparse Vector Representation: Treats financial statements as vectors where each dimension corresponds to a unique financial account, ignoring semantic information in account labels.
- Multichannel Sequence Representation: Incorporates semantic information from account identifiers and their corresponding amounts as sequences.
- Natural Language Representation: Converts financial statements into a sentence format, using templates like "Tag = Amount," with options for raw or relative values.
- Classification Algorithms:
- Decision-Tree Based Algorithm (LightGBM): Used as a baseline, it leverages sparse vector representations and different numerical representations (raw, relative values, binary presence/absence).
- LLM-clf (LLM with Classification Head): Employs a pre-trained LLM (LLaMA3.1) with a linear layer and softmax for classification. The LLM's latent representation is used for prediction.
- LLM-gen (LLM with Generation Head): Utilizes the generative capabilities of LLMs (LLaMA3.1-instruct, FinLLaMA3) to predict industry sectors by generating text based on a prompt. This approach also allows for natural language explanations of predictions.
- Text-Numeric Transformer: A novel two-stage architecture that first uses a Sentence-BERT encoder for account identifiers and then a fusion module to combine these embeddings with numerical amounts. This fused embedding is then processed by a Transformer encoder.
Datasets
The primary dataset used is derived from publicly available U.S. company filings from the SEC's EDGAR database, covering quarterly financial statements from January 2008 to March 2024. The study focuses on data from the fiscal year 2023 to avoid seasonal patterns. After extensive preprocessing, including filtering by sector, balance sheet/income statement tags, and number of tags, and data imputation, the final dataset comprises 9,582 submissions from 2,861 companies. Industry sectors are classified using 4-digit SIC codes, with the target labels being 8 official industry sectors derived from the first two digits of the SIC Code.
Key Insights and Ideas
- Hybrid Representations are Key: Combining numerical data with textual identifiers and leveraging advanced language models significantly outperforms traditional numerical-only approaches.
- Relative Values Matter: Normalizing financial statement amounts by total assets (relative values) generally leads to better classification performance than using raw amounts, especially for models sensitive to numerical scale.
- LLMs Offer Explainability: The LLM-gen approach, which uses generative capabilities, not only performs well but also provides natural language explanations for its predictions, a crucial feature for auditors.
- Text-Numeric Fusion is Powerful: The proposed Text-Numeric Transformer, which effectively fuses textual and numerical information, achieved the highest classification performance, demonstrating the benefit of integrating both data types.
- Task-Specific Fine-tuning is Crucial: Zero-shot classification with LLMs on this specialized financial task yielded weak performance, highlighting the need for fine-tuning or task-specific training.
Key Results
- LightGBM Baseline: Achieved a maximum MCC of 0.69 on the test set using relative values representation.
- LLM-clf: Both LLaMA3.1 and FinLLaMA3 achieved an MCC of 0.64 on the test set using relative values. Performance dropped significantly when amounts were excluded.
- LLM-gen: Achieved a competitive MCC of 0.66 on the test set with FinLLaMA3 using relative values. Notably, this model performed comparably well even with raw values, suggesting better handling of numerical data.
- Text-Numeric Transformer: This novel architecture, particularly with the FinLang model, achieved the highest MCC of 0.71 on the test set, outperforming all other methods. Experiments without amounts showed a significant performance drop, reinforcing the importance of numerical data.
Limitations and Future Work
- Imbalanced Dataset: The dataset's inherent imbalance posed challenges, and oversampling minority classes reduced performance. Future work could explore GANs and Diffusion Models for better imbalance handling.
- LLM Explanation Evaluation: While LLM-gen explanations were found to be coherent, a quantitative evaluation was difficult due to the lack of a grounded reference for sector-specific rules. Developing such datasets is a future direction.
- Larger LLMs: Due to GPU limitations, experiments with larger LLMs (e.g., LLaMA 405B) and advanced text-numeric models were not conducted, but are hypothesized to yield even better results.
- Broader Applications: The authors suggest extending their approach to other accounting tasks like anomaly detection and risk scoring, and refining data representations for more complex classification. The ultimate goal is to contribute to semi-automated auditing systems.
Summary updated