ITER: Iterative Transformer-based Entity Recognition and Relation Extraction

Moritz Hennen, Florian Babl, Michaela GeierhosEMNLP 2024 · DOI

Introduction

In the field of Natural Language Processing (NLP), extracting structured information—identifying entities and their relationships—is a foundational task. Recently, the field has transitioned toward autoregressive models (e.g., T5, GPT) that treat information extraction as a sequence-to-sequence translation task. While flexible and accurate, these models are computationally expensive and slow due to their token-by-token generation process, which limits parallelization.

Researchers from the University of the Bundeswehr Munich and Ludwig Maximilian University of Munich have introduced ITER (Iterative Transformer-based Entity Recognition and Relation Extraction) to address these efficiency bottlenecks. ITER demonstrates that an encoder-only transformer architecture can achieve state-of-the-art (SOTA) performance without the slow sequential decoding step, running up to 23 times faster than leading generative models.

Methods: The Iterative Pipeline

ITER replaces the text-generation process with a three-step iterative classification pipeline built on a standard transformer encoder (such as FLAN-T5 or DeBERTa). This approach allows for high degrees of parallelization across the sequence dimension.

The ITER three-step iterative classification pipeline.

The pipeline consists of three primary stages:

  1. is_left (Identify Starts): The model identifies all positions where a named entity span begins.
  2. is_span (Identify Ends): For each identified start position, the model predicts the most likely end position and the corresponding entity type (e.g., Person, Location). This design inherently handles nested entities by allowing multiple spans to overlap.
  3. is_link (Identify Relations): The model evaluates pairs of identified entities to predict whether a relationship exists between them (e.g., "Works For").

By optimizing the is_span step to look only at a fixed window of preceding markers, the model maintains linear time complexity relative to input length.

Experimental Setup

The model was evaluated on six diverse benchmarks covering various domains:

  • General News: ACE05 and CoNLL04.
  • Biomedical: ADE (drug-drug interactions) and GENIA (nested biological entities).
  • Scientific: SciERC (scientific abstracts).
  • NER Standards: CoNLL03.

Key Results

ITER achieves competitive or superior performance compared to much larger generative models while maintaining significantly higher throughput.

Performance comparison of ITER against state-of-the-art models across multiple benchmarks.

Performance Highlights

  • State-of-the-Art Accuracy: ITER set new SOTA records for Relation Extraction on ACE05 (71.9 F1) and ADE (85.6 F1), outperforming models with billions of parameters.
  • Inference Speed: On an NVIDIA RTX 4090, ITER reached throughputs of 605 samples/second on CoNLL04. In contrast, the autoregressive ASP model achieved only ~27 samples/second on the same hardware.
  • Efficiency: The authors found that the encoder stack from FLAN-T5 is highly effective for these discriminative tasks, allowing ITER to outperform models like ASP 3B with only ~400M parameters.

Conclusion and Future Work

ITER proves that the "decoder" is optional for high-performance information extraction. By utilizing an encoder-only setup with specialized heads, the model achieves SOTA results with a fraction of the latency of generative approaches. Future research may explore the model's performance on larger synthetic datasets to test extreme entity nesting and investigate zero-shot transfer capabilities across domains.

Summary updated

BibTeX
@inproceedings{hennen2024iter,
  author = {Moritz Hennen and Florian Babl and Michaela Geierhos},
  title = {{ITER: Iterative Transformer-based Entity Recognition and Relation Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.655},
}
Figure