Reflective Agreement: Combining Self-Mixture of Agents with a Sequence Tagger for Robust Event Extraction

Fatemeh Haji, Mazal Bethany, Cho-Yu Jason Chiang, Anthony Rios, Peyman NajafiradEMNLP 2025

TL;DR

This paper introduces ARIS, a novel hybrid approach for event extraction that combines the strengths of discriminative sequence taggers and generative Large Language Models (LLMs). By leveraging a "Self Mixture of Agents" for diverse predictions, agreement detection, confidence filtering, and a unique "reflective inference" mechanism, ARIS significantly improves the accuracy and robustness of event extraction, especially for complex and rare events.

What Problem Does This Paper Address?

Event Extraction (EE) is crucial for understanding unstructured text, but it's a challenging task. Traditional discriminative models excel at precision but struggle with recall, especially for less common events. On the other hand, generative LLMs offer better recall and flexibility but often suffer from hallucinations and inconsistent outputs due to their inherent stochasticity. Existing hybrid approaches, like multi-agent debates, can be computationally expensive and lack systematic ways to resolve disagreements. This paper aims to bridge this gap by creating a more robust and reliable event extraction system that combines the best of both worlds.

Methods

ARIS is built around a few key innovations:

  • Decomposed Instruction Fine-Tuning: The LLM is first fine-tuned on a specially crafted dataset that breaks down event extraction into smaller, manageable subtasks (trigger detection, classification, argument identification, etc.). This curriculum-based approach helps the LLM build a strong understanding of the entire event extraction pipeline.
  • Self Mixture of Agents (Self-MoA): Multiple instances of the fine-tuned LLM are run in parallel with slightly different sampling parameters. This generates a diverse set of potential event predictions, increasing the chances of capturing nuanced events.
  • Reflective Agreement Mechanism: This is the core of ARIS. It involves:
  • Consensus Detection: Identifying events where both the LLM agents and a discriminative sequence tagger (TagPrime) agree. These are considered the most reliable predictions.
  • Confidence Filtering: For disagreements, predictions with high confidence from either the sequence tagger or the LLM ensemble are retained. Low-confidence disagreements are discarded.
  • Reflection Module: Ambiguous disagreements (those not resolved by confidence filtering) are fed into a specialized LLM module. This module uses structured prompts to "reflect" on the context and the ambiguous predictions, making a final decision on the correct event or argument. This reflection process is crucial for resolving complex cases.

Datasets

The authors evaluated ARIS on three standard event extraction benchmarks:

  • CASIE: Covers cybersecurity news with 5 event types.
  • M2E2: Features shorter news content with 8 event types.
  • MLEE: Focuses on the biomedical domain with 29 event types.

They used the "split1" partitions from the TextEE benchmark for consistency.

Key Insights and Ideas

  • Decomposition is Key: Breaking down complex tasks like event extraction into smaller, teachable subtasks for LLMs significantly improves their performance, especially when combined with a structured learning curriculum.
  • Agreement as a Signal of Reliability: When both generative and discriminative models agree on an event, it's a strong indicator of accuracy. This consensus is a cornerstone of ARIS.
  • Reflection for Ambiguity Resolution: The "reflective inference" module is a novel way to handle cases where simple agreement or confidence filtering isn't enough. By using LLMs to reason about ambiguous predictions, ARIS can achieve higher accuracy.
  • Robustness to LLM Variability: ARIS demonstrates remarkable stability in performance even when LLM sampling temperatures vary, highlighting the effectiveness of its agreement and reflection mechanisms in normalizing diverse outputs.

Key Results

ARIS consistently outperformed existing state-of-the-art methods across all three benchmark datasets, particularly in argument extraction tasks. For instance, on the M2E2 dataset, ARIS achieved the strongest performance across all metrics. Notably, the ARIS approach with the Phi-3 model improved argument classification F1 scores over the strong TagPrime baseline by over 10 points. The ablation study also confirmed that ARIS effectively leverages the complementary strengths of discriminative and generative models.

Limitations and Future Work

The authors acknowledge that ARIS comes with significant computational overhead due to running multiple LLM instances for the Self-MoA component and the intensive training process for decomposed fine-tuning. The reflection mechanism, while effective, may still struggle with extremely challenging cases. Future work could explore cross-domain generalization through joint training across datasets.

Summary updated

BibTeX
@inproceedings{haji2025reflective,
  author = {Fatemeh Haji and Mazal Bethany and Cho-Yu Jason Chiang and Anthony Rios and Peyman Najafirad},
  title = {{Reflective Agreement: Combining Self-Mixture of Agents with a Sequence Tagger for Robust Event Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}

Loading related content...

Figure