Financial Product Ontology Population with Large Language Models

Chanatip Saetia, Jiratha Phruetthiset, Tawunrat Chalothorn, Monchai Lertsutthiwong, Supawat Taerungruang, Pakpoom BuabthongACL 2024

TL;DR

This paper explores how Large Language Models (LLMs) can automate the extraction of structured information from unstructured Thai financial documents to populate ontologies. They found that LLMs, particularly with carefully crafted prompts including examples, significantly outperform traditional methods, achieving a 61.05% F1 score in this challenging low-resource language domain.

What Problem Does This Paper Address?

In today's data-rich world, efficiently managing and understanding vast amounts of text is crucial. Ontologies provide a structured way to represent knowledge, enabling better interoperability and reasoning. However, populating these ontologies, especially in specialized domains like retail banking and in low-resource languages like Thai, is a significant challenge. Unstructured documents often lack consistent formatting, contain fragmented information, and require extensive manual effort for annotation. This paper tackles the problem of automatically extracting structured data from Thai corporate banking documents to enrich domain-specific ontologies, aiming to overcome data scarcity and linguistic complexity issues.

Methods

The researchers compared two main approaches:

  • Traditional Span-based Approach: This method uses a BIO (Beginning, Inside, Outside) tagging scheme, similar to Named Entity Recognition (NER). They fine-tuned a Thai pre-trained language model, Wangchanberta, on their dataset. A few-shot setting was also explored, providing a small number of examples to the model.
  • LLM-based Generative Approach: This approach leverages GPT-3.5 to directly generate structured output (JSON format) from text. They experimented with various prompting techniques:
  • Zero-shot: Providing only a task description.
  • Few-shot Prompting: Including positive and/or negative examples in the prompt to guide the model.
  • Chain-of-Thought (CoT) Prompting: Encouraging the model to reason step-by-step.
  • Definition from Schema.org: Incorporating definitions of fields from schema.org to provide explicit context.

They combined these prompting strategies to find the optimal configuration.

Datasets

The study used internal corporate documents detailing 20 retail banking products, specifically credit cards. These documents were manually tagged by a Thai linguist and verified by computational linguists. The ontology used aligns with Schema.org's PaymentCard concept, focusing on properties like floorLimit, monthlyMinimumRepaymentAmount, annualPercentageRate, and availableChannel. No new benchmark datasets were introduced; existing internal data was utilized.

Key Insights and Ideas

  • LLMs Shine in Low-Resource Settings: The study strongly suggests that LLMs are a powerful tool for ontology population, especially in languages with limited annotated data, like Thai.
  • Prompt Engineering is Key: The performance of LLMs is highly sensitive to how they are prompted. Simple adjustments can lead to significant improvements.
  • Examples are Highly Effective: Providing even a few examples (few-shot learning) in the prompt dramatically improves the LLM's ability to extract relevant information, particularly recall. Including both positive and negative examples was found to be especially beneficial for comprehension.
  • Contextual Definitions Boost Performance: Incorporating definitions from schema.org helps the LLM understand the specific meaning of fields, leading to more accurate extraction.
  • Traditional Methods Struggle: The traditional span-based approach, even with a fine-tuned model, showed very limited success, highlighting the limitations of older methods in this context.

Key Results

  • The traditional span-based approach achieved a low F1 score of 4.92% when fine-tuned. Even in a few-shot setting without class classification, it reached only 21.01%.
  • The LLM-based generative approach, with a basic zero-shot prompt, achieved 20.14% F1 score.
  • By incorporating few-shot learning (positive and negative examples), the F1 score improved significantly to 58.54%.
  • The best performing configuration, combining few-shot learning, CoT prompting, and schema.org definitions, achieved an impressive 61.05% F1 score. This represents a substantial improvement over traditional methods.

Limitations and Future Work

Limitations:

  • GPT-3.5 Generalization: The study used GPT-3.5, which is a general-purpose model. More specialized models might yield even better results.
  • Thai Language Nuances: Thai, as a low-resource language, inherently presents challenges, leading to lower performance compared to high-resource languages.
  • Annotation Subjectivity: Identifying entity spans can sometimes be subjective, depending on the entity type (e.g., a single number for floorLimit vs. a phrase for serviceLocation).
  • GPT-4 Output Style: While GPT-4 showed promise, its outputs were sometimes paraphrased or included extra information, requiring additional processing.

Future Work:

  • Extend to Other Domains: Apply the LLM-based approach to financial domains beyond banking and leverage Schema.org's broader schema.
  • Knowledge Graph Integration: Utilize the generated knowledge graphs for downstream applications like question answering or information retrieval.
  • Explore Other LLMs: Investigate the performance of other LLMs, including more advanced or specialized models.
  • Refine GPT-4 Prompting: Develop more restrictive prompting strategies for GPT-4 to prevent the generation of extraneous information.

Summary updated

BibTeX
@inproceedings{saetia2024financial,
  author = {Chanatip Saetia and Jiratha Phruetthiset and Tawunrat Chalothorn and Monchai Lertsutthiwong and Supawat Taerungruang and Pakpoom Buabthong},
  title = {{Financial Product Ontology Population with Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure