FinTruthQA: A Benchmark Dataset for Evaluating the Quality of Financial Information Disclosure

Ziyue Xu, Peilin Zhou, Xinyu Shi, Jiageng Wu, Yikang Jiang, Dading Chong, Bin Ke, Jie YangSubmission to ICLR 2025 · Rejected

TL;DR

FinTruthQA is a newly introduced benchmark of 6,000 expert-annotated Chinese financial Q&A pairs designed to automatically assess corporate disclosure quality across four criteria: question identification, question relevance, answer readability, and answer relevance. Extensive benchmarking reveals that while standard language models effortlessly solve question-level filtering, evaluating answer relevance remains notoriously difficult—where domain-adapted models like FinBERT outperform advanced zero-shot Large Language Models (LLMs) such as GPT-4.

What Problem Does This Paper Address?

In capital markets, retail investors frequently interact with listed firms via official exchange-hosted Q&A platforms (such as those hosted by the Shanghai and Shenzhen Stock Exchanges). While firms are legally required to provide accurate disclosures, responses often deflect, remain vague, or provide zero substantive information.

Evaluating the quality—rather than merely the volume—of corporate disclosures at scale is an open challenge in financial accounting. Existing Natural Language Processing (NLP) benchmarks primarily focus on financial sentiment analysis or numerical question-answering over financial tables, leaving automated auditing of real-world disclosure quality largely unaddressed.

Methods

The authors frame disclosure evaluation around four accounting-informed classification tasks:

  1. Question Identification: Binary detection of whether user input is a genuine query vs. a suggestion or complaint.
  2. Question Relevance: Binary detection of whether the query relates to corporate performance and investment topics.
  3. Answer Readability: 4-point ordinal scale measuring clarity, coherence, and lack of ambiguity.
  4. Answer Relevance: 3-point scale evaluating whether the response completely dodges (Level 1), partially answers (Level 2), or directly answers (Level 3) the investor's questions.

To benchmark these tasks, the authors evaluate:

  • Traditional Machine Learning: TF-IDF with Logistic Regression, SVM, Random Forest, and KNN.
  • Pre-trained Language Models (PLMs): General and financial domain models (BERT, RoBERTa, FinBERT, Mengzi-fin, DKPLM, SBERT-nli).
  • Continued Pre-Training (CPT): Domain-adapting select PLMs via Masked Language Modeling (MLM) using over 900,000 scraped financial Q&A pairs, incorporating custom prefix tokens ([Q] and [A]).
  • Zero-Shot LLMs: Evaluating GPT-4, LLaMA-3.1-8B, Qwen2-7B, and Mistral-7B using direct prompt scoring and Chain-of-Thought (CoT) prompting.
  • Quality Control via Confident Learning: Applying a model-in-the-loop audit on the subjective answer relevance labels to systematically flag and correct label noise.

Datasets

  • FinTruthQA (New Benchmark): Contains 6,000 manually annotated real-world Q&A pairs split equally between the Shanghai Stock Exchange (Shangzhen E-Hudong) and the Shenzhen Stock Exchange (Hudongyi).
  • Unlabeled Pre-training Corpus: Over 900,000 raw financial Q&A pairs collected via web scraping to support continued pre-training.

Key Insights and Ideas

  • Question-level vs. Answer-level Disparity: Question identification and relevance are largely "solved" by standard models (F1 > 98%), but evaluating corporate answers requires deep domain context and pragmatic reasoning, leading to much lower performance across all model families.
  • Domain Pre-training Beats General Scale: Supervised fine-tuning of domain-specific PLMs substantially outperformed general-purpose, large-scale LLMs. FinBERT with continued pre-training achieved the highest performance on answer relevance (Micro F1 80.04%, QWK 67.89), outperforming zero-shot GPT-4 (Micro F1 67.39%, QWK 59.06).
  • Chain-of-Thought (CoT) Degradation in Zero-Shot LLMs: Adding CoT prompts to GPT-4 decreased answer relevance performance (Micro F1 dropped from 67.39% to 59.76%). In Chinese financial contexts, reasoning step-by-step introduced domain hallucinations and keyword-matching fallacies rather than logical precision.
  • Common Failure Modes: LLMs frequently fail when distinguishing non-answers that borrow query keywords (giving a false sense of relevance), handling subtle entity/pronoun shifts (e.g., answering a question about "benefits" with details about "salaries"), or recognizing that stating "the situation is unclear" can sometimes be a legitimate, valid response.

Key Results

  • Question Tasks: On Question Identification, BERT achieved an accuracy of 96.89% and F1 of 98.32%, beating traditional TF-IDF + Logistic Regression (93.70% accuracy). For Question Relevance, BERT reached an accuracy of 99.67% and F1 of 99.85%.
  • Answer Readability: FinBERT led all models with 87.57% Micro F1 and 66.53 Quadratic Weighted Kappa (QWK), effectively identifying top-quality answers but struggling to differentiate subtle gradations between lower scores.
  • Answer Relevance:
  • Baseline ML (TF-IDF + LR): Micro F1 of 69.61%, QWK of 38.16.
  • Standard PLM (Mengzi-fin): Macro F1 of 68.96%, QWK of 66.31.
  • CPT-Enhanced (FinBERT + CPT): Best overall with 80.04% Micro F1 and 67.89 QWK.
  • Zero-shot LLMs: GPT-4 Direct Score achieved 67.39% Micro F1, outperforming open-weight models like Qwen2 (60.52%) and LLaMA-3.1 (53.00%), yet trailing specialized BERT models.

Limitations and Future Work

  • Jurisdiction Specificity: The dataset represents Chinese stock market interactions and regulations; findings may not directly generalize to regulatory disclosure frameworks in other countries.
  • Subjectivity & Class Imbalance: Certain categories exhibit heavy real-world imbalances (e.g., >99% of user inquiries were deemed relevant), which challenges model training and evaluation metrics.
  • LLM Adaptation: LLM evaluations were strictly zero-shot. Exploring parameter-efficient fine-tuning (PEFT), few-shot prompting, and retrieval-augmented generation (RAG) represents a logical next step to narrow the gap between LLMs and supervised domain PLMs.

Summary updated

Abstract

Accurate and transparent financial information disclosure is crucial in the fields of accounting and finance, ensuring market efficiency and investor confidence. Among many information disclosure platforms, the Chinese stock exchanges' investor interactive platform provides a novel and interactive way for listed firms to disclose information of interest to investors through an online question-and-answer (Q\&A) format. However, it is common for listed firms to respond to questions with limited or no substantive information, and automatically evaluating the quality of financial information disclosure on large amounts of Q\&A pairs is challenging. This paper builds a benchmark FinTruthQA, that can evaluate advanced natural language processing (NLP) techniques for the automatic quality assessment of information disclosure in financial Q\&A data. FinTruthQA comprises 6,000 real-world financial Q\&A entries and each Q\&A was manually annotated based on four conceptual dimensions of accounting: *question identification*, *question relevance*, *answer readability*, and *answer relevance*. We benchmarked various NLP techniques on FinTruthQA, including statistical machine learning models, pre-trained language model and their fine-tuned versions, as well as large language models (LLMs). Experiments showed that existing NLP models have strong predictive ability for *question identification* and *question relevance* tasks, but are suboptimal for *answer readability* and *answer relevance* tasks. By establishing this benchmark, we provide a robust foundation for the automatic evaluation of information disclosure, significantly enhancing the transparency and quality of financial reporting. FinTruthQA can be used by auditors, regulators, and financial analysts for real-time monitoring and data-driven decision-making, as well as by researchers for advanced studies in accounting and finance, ultimately fostering greater trust and efficiency in the financial markets.
BibTeX
@inproceedings{xu2026fintruthqa,
  author = {Ziyue Xu and Peilin Zhou and Xinyu Shi and Jiageng Wu and Yikang Jiang and Dading Chong and Bin Ke and Jie Yang},
  title = {{FinTruthQA: A Benchmark Dataset for Evaluating the Quality of Financial Information Disclosure}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaaiss.v9i1.42926},
}
Figure