Knowledge-augmented Financial Market Analysis and Report Generation

Yuemin Chen, Feifan Wu, Jingwei Wang, Hao Qian, Ziqi Liu, Zhiqiang Zhang, Jun Zhou, Meng Wang

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in text generation, yet they often falter when applied to the rigorous domain of financial market analysis. Crafting a professional market report requires more than just summarizing news; it necessitates the ability to connect disparate economic indicators—such as understanding how corporate demand deposits influence interbank liquidity—and synthesizing these into a logical narrative.

Current LLMs face three primary challenges in this field: frequent hallucinations, significant knowledge gaps regarding up-to-date financial expertise, and reasoning failures when performing multi-step logical chaining. While standard Retrieval-Augmented Generation (RAG) can provide relevant documents, financial data is often too noisy for simple text search to offer the structured logical scaffolding required for effective reasoning.

Comparison of analysis quality between vanilla LLMs, document-based RAG, and the proposed FKG-based retrieval system.

The TFR Framework: Methodology

To address these limitations, researchers proposed the Two-stage FKG-based Retrieval (TFR) framework. This system grounds LLMs in structured logical relationships by utilizing a Financial Knowledge Graph (FKG). The framework operates in three distinct phases:

  1. LLM-based KG Construction: The system uses LLMs to extract "triples" (Subject-Predicate-Object) from a vast corpus of expert reports. This process maps out causal relationships, such as "Interest Rate -> affects -> Bond Yield," creating a structured map of financial logic.
  2. Clustering-based Triple Retrieval: To manage the inherent messiness of financial terminology, the authors employ an agglomerative clustering algorithm on node embeddings. This ensures that the retrieved knowledge is diverse and semantically relevant, preventing the model from becoming trapped in a single "semantic neighborhood."
  3. Two-Stage RAG Process:
    • Stage 1 (Fact Selection): The FKG is used to navigate noisy data and select only the facts most relevant to the query.
  4. Stage 2 (Reasoning & Generation): The model integrates these selected facts with retrieved logical triples to "reason out" the final analysis.
The architectural overview of the TFR framework, comprising KG extraction, cluster-based retrieval, and two-stage RAG.

Furthermore, the researchers utilized Instruction Fine-Tuning (SFT) on a 6B-parameter model (GLM3-6b). This training taught the model to specifically expect and utilize structured KG triples, ensuring a seamless integration of logic into its writing style.

The FMAG Benchmark

A critical contribution of this work is the introduction of the Financial Market Analysis Generation (FMAG) benchmark. Derived from 6,000 expert-written analyst reports on the Chinese bond market, the dataset includes 2,188 training and 295 test instances. Each instance pairs a question with raw financial facts and a reference analysis, requiring models to synthesize numerical data into a cohesive, expert-level narrative.

Key Results and Insights

The TFR framework was evaluated using metrics including GLM4-Score (for logical consistency), BERTScore, and RougeL. The results demonstrated a significant performance leap:

  • Enhanced Logical Consistency: The proposed GLM3-6b + SFT + TFR model achieved a consistency score of 3.2203, outperforming the vanilla GLM3-6b (2.7424) and even surpassing GPT-3.5-turbo in specific reasoning tasks.
  • Synergy of SFT and RAG: The study found that models specifically fine-tuned for KG integration performed best. Interestingly, models trained without this integration saw performance decreases when KG triples were added, highlighting the importance of specialized training.
  • Efficiency of Small Models: A specialized 6B-parameter model using the TFR framework can outperform much larger general-purpose LLMs in domain-specific reasoning, suggesting a more efficient path for specialized AI applications.

Limitations and Future Directions

While promising, the framework's performance is currently bottlenecked by the quality of the initial Knowledge Graph; errors in triple extraction can propagate through the system. Additionally, the automatically constructed KG can contain redundant nodes for similar phrases, complicating logical pathfinding. Future work aims to expand the benchmark beyond the bond market into more volatile sectors like equities and commodities, where causal relationships are even more complex.

Summary updated

Abstract

Crafting a convincing financial market analysis report necessitates a wealth of market information and the expertise of financial analysts, posing a highly challenging task. While large language models (LLMs) have enabled the automated generation of financial market analysis text, they still face issues such as hallucinations, errors in financial knowledge, and insufficient capability to reason about complex financial problems, which limits the quality of the generation. To tackle these shortcomings, we propose a novel task and a retrieval-augmented framework grounded in a financial knowledge graph (FKG). The proposed framework is compatible with commonly used instruction-tuning methods. Experiments demonstrate that our framework, coupled with a small-scale language model fine-tuned with instructions, can significantly enhance the logical consistency and quality of the generated analysis texts, outperforming both large-scale language models and other retrieval-augmented baselines.
BibTeX
@inproceedings{chen2024knowledge,
  author = {Yuemin Chen and Feifan Wu and Jingwei Wang and Hao Qian and Ziqi Liu and Zhiqiang Zhang and Jun Zhou and Meng Wang},
  title = {{Knowledge-augmented Financial Market Analysis and Report Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-industry.90},
}
Figure