Financial Named Entity Recognition: How Far Can LLM Go?

Yingdong Lu, Yintong Huo

Abstract

The surge of large language models (LLMs) has revolutionized the extraction and analysis of crucial information from a growing volume of financial statements, announcements, and business news. Recognition for named entities to construct structured data poses a significant challenge in analyzing financial documents and is a foundational task for intelligent financial analytics. However, how effective are these generic LLMs and their performance under various prompts are yet need a better understanding. To fill in the blank, we present a systematic evaluation of state-of-the-art LLMs and prompting methods in the financial Named Entity Recognition (NER) problem. Specifically, our experimental results highlight their strengths and limitations, identify five representative failure types, and provide insights into their potential and challenges for domain-specific tasks.

Introduction

Named Entity Recognition (NER)—the task of identifying and categorizing entities like organizations, people, and locations—is the bedrock of financial data structuring. However, financial text is notoriously nuanced. A generic model might see the phrase "Johnson Brothers rethink plan..." and label "Johnson Brothers" as a Person based on the name, whereas in a financial context, it is clearly an Organization.

While specialized, fine-tuned models (like BERT variants) have traditionally handled this, the industry is shifting toward general-purpose LLMs. This study addresses a critical gap: understanding how these generic LLMs perform across different prompting strategies in the financial domain and establishing a taxonomy of their failure modes.

Methodology

The researchers conducted a systematic evaluation comparing state-of-the-art LLMs against traditional fine-tuned baselines.

Models Evaluated

  • LLMs: GPT-4o (and mini), LLaMA-3.1 (70B and 8B), and Gemini-1.5-Flash (and an 8B version).
  • Baselines: Fine-tuned BERT and RoBERTa models.

Prompting Strategies

The study explored three primary prompting techniques to evaluate zero-shot and few-shot capabilities:

  1. Direct Prompting: A simple zero-shot instruction to identify entities.
  2. In-Context Learning (ICL): Providing the model with five randomly selected examples (5-shot) to establish pattern recognition.
  3. Chain-of-Thought (CoT): Using the "let’s think step by step" instruction to encourage the model to explain its reasoning before providing the final labels.
Example of a Direct Prompt used for financial NER.

Dataset: FiNER-ORD

The study utilizes the FiNER-ORD dataset, consisting of 201 manually annotated financial news articles. Unlike older datasets (like CRA) which are heavily skewed toward certain entity types, FiNER-ORD provides a more balanced distribution of Organizations (ORG), Locations (LOC), and Persons (PER). This makes it a robust benchmark for real-world financial applications where diverse entity types are common.

Key Insights and Performance Analysis

The evaluation revealed several critical findings regarding the behavior of LLMs in specialized domains:

  • The "Overthinking" Trap: Surprisingly, Chain-of-Thought (CoT) prompting isn't always beneficial. For the LLaMA-3.1 series, CoT actually degraded performance. The models began identifying "implied entities"—tagging words that merely suggested an entity rather than the entity itself.
  • Few-Shot is the Sweet Spot: Moving from zero-shot to 5-shot (ICL) provided the most significant performance boost across all LLMs, narrowing the gap with fine-tuned models.
  • Gemini's Superiority: In the few-shot and CoT categories, the Gemini-1.5 series consistently outperformed both GPT-4o and LLaMA-3.1, showing particular strength in handling the organizational nuances of financial text.
  • The Persistence of Fine-Tuning: Despite the massive scale of LLMs, smaller models like RoBERTa, when specifically fine-tuned on financial data, still maintain the highest overall accuracy (F1 scores).

Experimental Results

Performance was measured using the Weighted F1 score, which accounts for the frequency of different entity types across the dataset.

Comprehensive performance comparison of fine-tuned models and LLMs across different prompting strategies.

As shown in the data above:

  • Fine-tuned RoBERTa achieved the gold standard with a score of 0.8792.
  • Best Performing LLM: Gemini-1.5 (with CoT) reached 0.8369, showing that LLMs are close but not yet superior to specialized small models.
  • Prompting Impact: For GPT-mini, moving from Direct Prompting to In-Context Learning jumped the score from 0.7396 to 0.7743.

Failure Analysis

The authors identified five main failure modes. Contextual Misunderstanding (e.g., confusing a company name for a person) was the most common, accounting for 31.3% of errors. Other errors included boundary detection issues and over-reasoning.

Conclusion and Future Directions

While LLMs are powerful, their "generic" nature is a double-edged sword in specialized fields like finance. The study suggests two primary paths forward:

  1. Domain-Specific Tuning: Even for LLMs, a layer of financial-specific fine-tuning is likely necessary to overcome contextual misunderstanding errors.
  2. Self-Correction Mechanisms: Developing prompts that allow the model to "verify" its own labels could mitigate common boundary and classification errors.

The study concludes that while LLMs are rapidly improving, the financial sector should continue to leverage fine-tuned BERT-style models for tasks requiring high precision.

Summary updated

Abstract

The surge of large language models (LLMs) has revolutionized the extraction and analysis of crucial information from a growing volume of financial statements, announcements, and business news. Recognition for named entities to construct structured data poses a significant challenge in analyzing financial documents and is a foundational task for intelligent financial analytics. However, how effective are these generic LLMs and their performance under various prompts are yet need a better understanding. To fill in the blank, we present a systematic evaluation of state-of-the-art LLMs and prompting methods in the financial Named Entity Recognition (NER) problem. Specifically, our experimental results highlight their strengths and limitations, identify five representative failure types, and provide insights into their potential and challenges for domain-specific tasks.
BibTeX
@inproceedings{lu2025financial,
  author = {Yingdong Lu and Yintong Huo},
  title = {{Financial Named Entity Recognition: How Far Can LLM Go?}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2025},
}
Figure