FinTruthQA: A Benchmark Dataset for Evaluating the Quality of Financial Information Disclosure

Ziyue Xu, Peilin Zhou, Xinyu Shi, Jiageng Wu, Yikang Jiang, Bin Ke, Jie Yang

Abstract

Accurate and transparent financial information disclosure is crucial in the fields of accounting and finance, ensuring market efficiency and investor confidence. Among many information disclosure platforms, the Chinese stock exchanges' investor interactive platform provides a novel and interactive way for listed firms to disclose information of interest to investors through an online question-and-answer (Q&A) format. However, it is common for listed firms to respond to questions with limited or no substantive information, and automatically evaluating the quality of financial information disclosure on large amounts of Q&A pairs is challenging. This paper builds a benchmark FinTruthQA, that can evaluate advanced natural language processing (NLP) techniques for the automatic quality assessment of information disclosure in financial Q&A data. FinTruthQA comprises 6,000 real-world financial Q&A entries and each Q&A was manually annotated based on four conceptual dimensions of accounting. We benchmarked various NLP techniques on FinTruthQA, including statistical machine learning models, pre-trained language model and their fine-tuned versions, as well as the large language model GPT-4. Experiments showed that existing NLP models have strong predictive ability for real question identification and question relevance tasks, but are suboptimal for answer relevance and answer readability tasks. By establishing this benchmark, we provide a robust foundation for the automatic evaluation of information disclosure, significantly enhancing the transparency and quality of financial reporting.

Introduction

In the financial world, the prevailing assumption has often been that "more information" equates to "better information." However, traditional research has largely focused on the quantity of corporate disclosures—such as the volume of annual reports or press releases—while frequently overlooking the qualitative substance of these communications.

Recently, interactive platforms hosted by the Shanghai Stock Exchange (SSE) and Shenzhen Stock Exchange (SZSE) have emerged as vital channels for retail investors to engage directly with listed firms. While these platforms facilitate transparency, they are also prone to "fluff" or non-substantive responses from companies seeking to sidestep difficult inquiries. Until now, there has been no standardized method to automatically evaluate whether a company is providing meaningful disclosure or merely obfuscating. FinTruthQA addresses this gap by providing a benchmark for the automatic quality assessment of financial Q&A data.

The FinTruthQA Dataset

The core contribution of this work is the FinTruthQA dataset, which consists of 6,000 real-world Q&A entries sourced from the SSE and SZSE investor platforms. To ensure high-quality benchmarking, each entry was manually annotated by experts across four critical dimensions of accounting quality:

  1. Question Identification: Distinguishing genuine inquiries from complaints or suggestions.
  2. Question Relevance: Determining if the question pertains to investment-related topics.
  3. Answer Readability: Scoring the clarity and linguistic quality of the response (on a scale of 1–4).
  4. Answer Relevance: Assessing how directly the answer addresses the specific question (on a scale of 1–3).
Distribution of question and answer lengths in the FinTruthQA dataset.

The dataset reflects real-world distributions, including imbalanced classes where irrelevant questions are significantly less frequent than relevant ones, providing a realistic challenge for automated models.

Methodology

The researchers implemented a multi-stage pipeline to evaluate disclosure quality, testing three distinct categories of models: traditional machine learning (e.g., Logistic Regression, Random Forest), Pre-trained Language Models (PLMs) like BERT and its financial variants (FinBERT), and Large Language Models (LLMs) including GPT-4, Llama-3, and Qwen2.

Overview of the FinTruthQA evaluation framework, model selection, and dataset construction pipeline.

A significant technical component of the methodology is Continual Pre-training (CPT). General-purpose models were further trained on a massive unlabeled corpus of over 900,000 financial Q&A pairs. This process allows models to internalize domain-specific jargon and the unique nuances of corporate financial discourse before being fine-tuned on the annotated FinTruthQA dataset.

Key Results and Insights

The benchmarking process revealed several critical insights into the current state of NLP in the financial domain:

  • The "Answer Gap": While models are highly proficient at identifying questions (achieving up to a 98.32% F1 score), their performance declines sharply when evaluating answer quality. Distinguishing between a "partially related" and a "fully responsive" answer remains a significant challenge for even the most advanced models.
  • The Power of Specialization: Supervised BERT-based models, particularly those that underwent continual pre-training, often outperformed zero-shot LLMs like GPT-4. This suggests that for niche, highly regulated domains like Chinese finance, smaller, domain-specific models can be more reliable than massive general-purpose ones.
  • Readability vs. Relevance: FinBERT achieved a 66.53% Quadratic Weighted Kappa (QWK) for answer readability, showing strong alignment with human judgment. However, answer relevance proved tougher, with the best-performing model (FinBERT with CPT) reaching a 70.58% Macro F1, while GPT-4 trailed at 63.11%.

Conclusion and Future Work

FinTruthQA demonstrates that high-frequency interaction on investor platforms does not inherently guarantee transparency. Automated tools are essential for holding firms accountable for the substance of their communications. While current models excel at filtering noise, judging the nuance of corporate responses requires further advancement.

Future research will explore the integration of Retrieval-Augmented Generation (RAG) to provide models with broader context from official financial reports, potentially bridging the performance gap in evaluating answer relevance and substance. By establishing this benchmark, the researchers provide a foundation for real-time monitoring and data-driven decision-making by auditors, regulators, and analysts.

Summary updated

Abstract

Accurate and transparent financial information disclosure is crucial in the fields of accounting and finance, ensuring market efficiency and investor confidence. Among many information disclosure platforms, the Chinese stock exchanges' investor interactive platform provides a novel and interactive way for listed firms to disclose information of interest to investors through an online question-and-answer (Q&A) format. However, it is common for listed firms to respond to questions with limited or no substantive information, and automatically evaluating the quality of financial information disclosure on large amounts of Q&A pairs is challenging. This paper builds a benchmark FinTruthQA, that can evaluate advanced natural language processing (NLP) techniques for the automatic quality assessment of information disclosure in financial Q&A data. FinTruthQA comprises 6,000 real-world financial Q&A entries and each Q&A was manually annotated based on four conceptual dimensions of accounting. We benchmarked various NLP techniques on FinTruthQA, including statistical machine learning models, pre-trained language model and their fine-tuned versions, as well as the large language model GPT-4. Experiments showed that existing NLP models have strong predictive ability for real question identification and question relevance tasks, but are suboptimal for answer relevance and answer readability tasks. By establishing this benchmark, we provide a robust foundation for the automatic evaluation of information disclosure, significantly enhancing the transparency and quality of financial reporting. FinTruthQA can be used by auditors, regulators, and financial analysts for real-time monitoring and data-driven decision-making, as well as by researchers for advanced studies in accounting and finance, ultimately fostering greater trust and efficiency in the financial markets.
BibTeX
@inproceedings{xu2026fintruthqa,
  author = {Ziyue Xu and Peilin Zhou and Xinyu Shi and Jiageng Wu and Yikang Jiang and Bin Ke and Jie Yang},
  title = {{FinTruthQA: A Benchmark Dataset for Evaluating the Quality of Financial Information Disclosure}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaaiss.v9i1.42926},
}
Figure