Semantic Frame Parsing for Information Extraction : the CALOR corpus

Gabriel Marzinotto, Jeremy Auguste, Frederic Bechet, Géraldine Damnati, Alexis NasrarXiv 2018 · arXiv

TL;DR

This paper introduces CALOR, a large French encyclopedic corpus designed for partial semantic frame parsing in information extraction tasks. By focusing on a domain-specific subset of frames rather than full-text parsing, the authors curate a dense dataset and demonstrate that an integrated multi-task bi-LSTM sequence labeling approach outperforms traditional multi-model Conditional Random Fields (CRFs) in identifying semantic roles.

What Problem Does This Paper Address?

Semantic frame parsing—identifying events ("Frames") and their participants ("Frame Elements") in text—is crucial for Natural Language Understanding. Traditionally, benchmarks like Berkeley FrameNet focus on full-text parsing, attempting to label every single word and frame in a sentence.

However, full-text parsing faces two major hurdles:

  1. High Cost & Sparsity: Annotating everything is incredibly expensive, resulting in corpora where many frames appear only a handful of times, creating a severe data-sparsity problem for machine learning models.
  2. Mismatched Application Goals: For practical applications like Information Extraction (IE), practitioners rarely care about every generic frame; they only need specific, domain-relevant insights.

The authors address this gap by advocating for partial frame parsing: annotating only a targeted, high-relevance subset of frames. This makes manual annotation much more affordable, allowing for larger, denser datasets that can better train robust machine learning models.

Figure 1: Example of Frame multilabel annotation

Methods

To parse frames and roles simultaneously, the authors flatten the hierarchical frame structures into a flat sequence labeling problem using a standard B,I,O (Begin, Inside, Outside) encoding scheme. They compare two distinct sequence labeling paradigms to handle the multi-label challenge (where a single word can belong to multiple frames):

  1. Multi-model CRF Approach: The authors train a separate CRF model for each word in the Lexical Unit (LU) lexicon. At inference time, the sentence is duplicated for each recognized LU, processed by its respective CRF, and the outputs are merged. This drastically reduces label ambiguity for each individual model.
  2. Multi-task bi-LSTM Approach: A single bidirectional LSTM architecture shares its underlying word and syntactic representations across all frames. However, it uses task-specific fully-connected output layers where each frame is treated as an independent task.

Because word-level sequence tagging can produce structural inconsistencies (e.g., a role filler missing its corresponding frame trigger), the authors apply a simple Coherence Filter as a post-processing step to enforce logical links between triggers and roles.

Figure 3: Two different strategies (CRF and bi-LSTM multitask) for Frame parsing

Datasets

The primary contribution is the CALOR corpus, a 1.3-million-word collection of French encyclopedic history texts. It aggregates documents from four distinct sources:

  • Wikipedia's Archaeology portal
  • Wikipedia's World War 1 portal
  • Vikidia's Prehistory and Antiquity portals (written for children aged 8–13)
  • ClioTexte (historical documents like letters and speeches)

CALOR focuses on 53 frames and 145 Lexical Units, yielding 26,725 frame occurrences and 57,688 Frame Element annotations. Compared to existing French frame corpora like ASFALDA, CALOR has a smaller frame lexicon but a significantly higher density of examples per frame.

Key Insights and Ideas

  • The Ambiguity vs. Data Fragmentation Trade-off: Splitting the dataset to train individual CRFs per word (the multi-model approach) successfully minimizes label ambiguity. However, it fragments the training data. The multi-task bi-LSTM wins out because it leverages the entire dataset globally, discovering shared syntactic and semantic patterns across different frames.
  • Precision vs. Recall Profiles: The two architectures exhibit completely different precision-recall behaviors. The multi-model CRF is highly precise but suffers from low recall because it cannot generalize across different lexical units. Conversely, the multi-task bi-LSTM achieves much higher recall by extrapolating patterns across the entire corpus, though it is more prone to precision errors.

Key Results

The models were evaluated across four cascading sub-tasks: Trigger Identification (TI), Trigger Classification (TC), Role Identification (RI), and Role Classification (RC).

At the Equal Error Rate (EER) operating point, the quantitative results show:

  • Trigger Tasks (TI & TC): The multi-model CRF performs slightly better, achieving 96.4% on TI and 95.2% on TC, compared to the bi-LSTM's 95.5% and 92.3%. This confirms that isolating the label space per word helps resolve trigger ambiguity.
  • Role Tasks (RI & RC): The multi-task bi-LSTM clearly dominates the more complex task of argument classification, achieving an F-measure of 63.2% on Role Classification (RC) compared to the CRF's 60.6%.
Precision-Recall curves highlighting the performance trade-offs on subtask 4

Limitations and Future Work

While the authors demonstrate the value of the multi-task bi-LSTM, its precision drops significantly when aiming for high recall. Additionally, the paper notes that the corpus was specifically designed to study domain and style dependencies (e.g., training on Wikipedia archaeology and testing on WW1 texts, or comparing children's encyclopedias to historical essays), but explicit cross-domain evaluation experiments are left for future work.

Summary updated

Abstract

This paper presents a publicly available corpus of French encyclopedic history texts annotated according to the Berkeley FrameNet formalism. The main difference in our approach compared to previous works on semantic parsing with FrameNet is that we are not interested here in full text parsing but rather on partial parsing. The goal is to select from the FrameNet resources the minimal set of frames that are going to be useful for the applicative framework targeted, in our case Information Extraction from encyclopedic documents. Such an approach leverages the manual annotation of larger corpora than those obtained through full text parsing and therefore opens the door to alternative methods for Frame parsing than those used so far on the FrameNet 1.5 benchmark corpus. The approaches compared in this study rely on an integrated sequence labeling model which jointly optimizes frame identification and semantic role segmentation and identification. The models compared are CRFs and multitasks bi-LSTMs.
BibTeX
@misc{marzinotto2018semantic,
  author = {Gabriel Marzinotto and Jeremy Auguste and Frederic Bechet and Géraldine Damnati and Alexis Nasr},
  title = {{Semantic Frame Parsing for Information Extraction : the CALOR corpus}},
  year = {2018},
  eprint = {1812.08039},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.08039},
}
Figure