JaCorpTrack: Corporate History Event Extraction for Tracking Organizational Changes

Yuya Sawada, Hiroki Ouchi, Yuichiro Yasui, Hiroki Teranishi, Yuji Matsumoto, Taro Watanabe, Masayuki IshiiEMNLP 2025 · OpenReview

Abstract

Corporate history in corporate annual reports includes events related to organizational changes, which can provide useful cues for a comprehensive understanding of corporate actions. However, extracting organizational changes requires identifying differences in companies before and after an event, raising concerns about whether existing information extraction systems can accurately capture the relations. This work introduces JaCorpTrack, a novel event extraction task designed to identify events related to organizational changes. JaCorpTrack defines five event types related to organizational changes and is designed to identify the company names before and after each event, as well as the corresponding date. Experimental results indicate that large language models (LLMs) exhibit notable disparities in performance across event types. Our analysis reveals that these systems face challenges in identifying company names before and after events, and in interpreting event types expressed under ambiguous terminology. We will publicly release our dataset and experimental code at https://github.com/naist-nlp/JaCorpTrack.

Introduction

Investors and financial analysts rely heavily on corporate annual reports to assess a company’s trajectory and health. A critical component of these reports is the "Corporate History" section, which provides a chronological timeline of significant milestones. However, tracking a company through decades of evolution—involving name changes, mergers, and spin-offs—presents a significant challenge for automated systems.

Existing Event Extraction (EE) tools are often optimized for short news snippets. In contrast, corporate histories require long-range temporal reasoning. For instance, to correctly identify the entities in a merger occurring in 2002, a system must account for a name change that may have occurred decades earlier in the text. This "detective work" across a 70-year timeline is the primary motivation behind the JaCorpTrack benchmark.

Overview of the JaCorpTrack event extraction task, showing the transformation from raw timeline text to structured event tables.

Methodology and Task Definition

The researchers formalize the extraction process as an event table filling task. Rather than simple keyword identification, the system must extract structured tuples comprising a date and the relevant company entities for specific event types.

Event Types

JaCorpTrack defines five core event types related to organizational changes:

  1. ESTABLISH: The founding of a company.
  2. CHANGE: Rebranding or legal name changes.
  3. MERGE: Integration of entities, distinguishing between absorbed and surviving companies.
  4. SPLIT: The spin-off of a business unit.
  5. LIQUIDATION: The formal dissolution of a company.
Definition of event types and their corresponding argument roles in the JaCorpTrack dataset.

The study evaluated two primary modeling approaches:

  • Traditional Baselines: BERT-based models (DMBERT and BERT+CRF) utilizing standard sequence labeling.
  • Large Language Models (LLMs): Llama-3 (8B) and GPT-4o. The researchers explored both In-context learning (ICL) and Supervised Finetuning (SFT) using LoRA for the Llama-3 model.

The JaCorpTrack Dataset

The dataset consists of 210 corporate histories sampled from EDINET, Japan’s electronic disclosure system. It encompasses nearly 10,000 timeline entries and over 4,200 annotated events. JaCorpTrack is characterized by its complexity; the average document length is significantly greater than previous financial EE datasets, and the distance between related information (e.g., a date and its associated company name) is much larger, testing the limits of model memory and context handling.

Key Insights and Experimental Results

The experimental evaluation yielded several critical insights into the capabilities of modern LLMs:

  • The "Memorization" Trap: To test if LLMs were truly extracting information or relying on pre-trained knowledge, the authors conducted an anonymization experiment (e.g., replacing "Sony" with "[COMPANY_1]"). While GPT-4o’s performance dropped significantly, the finetuned Llama-3 remained stable, suggesting that top-tier LLMs often leverage internal knowledge of famous corporations rather than pure textual extraction.
  • Distance and Complexity: Performance was highest for "ESTABLISH" events (~90 F1), where dates and names typically appear in close proximity. Conversely, "MERGE" and "SPLIT" events proved difficult due to the large textual distance between relevant entities.
  • Terminology Ambiguity: The "SPLIT" event was the most challenging (as low as 33.8 F1). Ambiguous terms like "separated" or "transferred" often refer to simple transactions rather than formal organizational splits, leading to high error rates.
  • Finetuning Efficacy: For smaller models like Llama-3 (8B), finetuning was transformative, raising the F1-score from 22.7 (prompting) to 75.6. GPT-4o achieved the highest overall F1-score of 78.5.

Conclusion and Future Directions

While LLMs show promise, significant gaps remain in handling long-range dependencies and ambiguous terminology in corporate histories. Future work will focus on expanding the dataset beyond Japanese reports and improving reasoning through advanced prompt engineering like Chain-of-Thought. Additionally, integrating these extractions into automated Knowledge Bases via Entity Linking—ensuring the system recognizes a single entity across various name changes over decades—remains a key objective.

Summary updated

Abstract

Corporate history in corporate annual reports includes events related to organizational changes, which can provide useful cues for a comprehensive understanding of corporate actions.However, extracting organizational changes requires identifying differences in companies before and after an event, raising concerns about whether existing information extraction systems can accurately capture the relations.This work introduces JaCorpTrack, a novel event extraction task designed to identify events related to organizational changes.JaCorpTrack defines five event types related to organizational changes and is designed to identify the company names before and after each event, as well as the corresponding date.Experimental results indicate that large language models (LLMs) exhibit notable disparities in performance across event types.Our analysis reveals that these systems face challenges in identifying company names before and after events, and in interpreting event types expressed under ambiguous terminology.We will publicly release our dataset and experimental code at https://github.com/naist-nlp/JaCorpTrack
BibTeX
@inproceedings{sawada2025jacorptrack,
  author = {Yuya Sawada and Hiroki Ouchi and Yuichiro Yasui and Hiroki Teranishi and Yuji Matsumoto and Taro Watanabe and Masayuki Ishii},
  title = {{JaCorpTrack: Corporate History Event Extraction for Tracking Organizational Changes}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=ZV3S8SVgHO},
}
Figure