SPEECH: Structured Prediction with Energy-Based Event-Centric Hyperspheres

Shumin Deng, Shengyu Mao, Ningyu Zhang, Bryan Hooi

Abstract

Event-Centric Structured Prediction (ECSP) involves the complex task of identifying event triggers, classifying their types, and extracting their inter-relationships across documents. Traditional methods often struggle with the manifold dependencies inherent in these structures. This paper introduces SPEECH, a novel framework that leverages Energy-Based Models (EBMs) and hyperspherical geometric representations to capture these dependencies. By optimizing energy functions across token, sentence, and document levels, SPEECH achieves state-of-the-art performance on major benchmarks, significantly outperforming both generative and discriminative baselines.

Introduction

In Natural Language Processing, events are rarely isolated. The identification of an event (Event Detection) is intrinsically linked to its participants and its relationship to other occurrences (Event Relation Extraction). For instance, identifying a "shooting" trigger is inextricably tied to identifying the "victim" and determining causal links to a subsequent "death." This interconnectedness defines the challenge of Event-Centric Structured Prediction (ECSP).

Existing approaches typically treat these components as independent tasks or rely on simplified graph structures that fail to capture the non-linear, high-dimensional dependencies spanning across sentences and documents. SPEECH addresses these limitations by modeling the entire event structure through a unified energy-based framework, representing event classes as geometric hyperspheres to better capture class-level nuances.

Methodology

Energy-Based Modeling (EBM)

Unlike standard classifiers that utilize a softmax layer to output probabilities, SPEECH employs energy functions. In this paradigm, the model assigns a scalar "energy" value to a pair of input text and a proposed event structure. The learning objective is to minimize the energy for the correct structure while maximizing it for incorrect ones. This approach provides mathematical flexibility, allowing the model to account for label dependencies without the constraints of traditional probability normalization.

Hyperspherical Representations

To represent event classes (e.g., Conflict, Justice, Personnel), SPEECH utilizes hyperspheres in a high-dimensional embedding space. Each event type is defined by a centroid and a radius. Event mentions are mapped into this space, and their compatibility with a specific class is determined by their proximity to the surface of that class's hypersphere. This geometric approach allows for more nuanced representation than traditional point-based prototypes, as it accounts for the variance and boundaries of different event types.

Hierarchical Energy Architecture

SPEECH operates across three hierarchical levels to ensure a comprehensive understanding of the document:

  1. Token-level energy: Captures long-range dependencies between individual words to accurately identify event triggers.
  2. Sentence-level energy: Utilizes the hypersphere model to categorize identified event mentions into specific, fine-grained types.
  3. Document-level energy: Models the complex interactions between pairs of event mentions, facilitating the extraction of temporal, causal, and coreference relations across the entire text.

Experimental Setup

Datasets

The framework was evaluated on two primary benchmarks:

  • MAVEN-ERE: A large-scale dataset featuring fine-grained annotations for event coreference, as well as temporal, causal, and subevent relations.
  • ONTOEVENT-DOC: A document-level reconstruction of the OntoEvent dataset, designed to support document-wide relation extraction tasks.

Baselines

SPEECH was compared against a variety of competitive baselines, including sequence labeling models like BERT-CRF and generative frameworks such as TANL and Text2Event.

Results and Discussion

SPEECH demonstrated a predominant performance advantage across all evaluated tasks. On the MAVEN-ERE trigger classification task, SPEECH achieved an F1 score of 79.09%, representing a significant improvement over the previous state-of-the-art, CorED-BERT (68.49%).

The Impact of Energy and Geometry

Ablation studies confirmed that the energy-based components are critical to the model's success. Removing the energy functions resulted in a ~3% drop in trigger F1 scores. Furthermore, the hyperspherical representation consistently outperformed prototype-based methods, suggesting that the geometric boundary provided by the hypersphere is more effective at capturing the distribution of event mentions than a single central point.

Visualization and Qualitative Analysis

Case studies and visualizations revealed that SPEECH successfully clusters event mentions near the surfaces of their respective hyperspheres. In contrast, models lacking the energy-based objective produced scattered and less distinct representations, leading to higher error rates in classification and relation extraction.

Conclusion and Future Work

SPEECH introduces a robust framework for capturing the complex web of event dependencies through energy-based hyperspheres. While the model shows superior accuracy, the authors note that the complexity of energy calculations across multiple tasks results in high computational intensity and slower training times. Additionally, like many deep learning models, it faces challenges with data imbalance in rare event types.

Future research will focus on optimizing the efficiency of the energy calculations and extending the hyperspherical approach to other structured prediction domains beyond event extraction.

Summary updated

BibTeX
@inproceedings{deng2023speech,
  author = {Shumin Deng and Shengyu Mao and Ningyu Zhang and Bryan Hooi},
  title = {{SPEECH: Structured Prediction with Energy-Based Event-Centric Hyperspheres}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.21},
}
Figure