Automatic Event Salience Identification
Abstract
Identifying the salience (i.e. importance) of discourse units is an important task in language understanding. While events play important roles in text documents, little research exists on analyzing their saliency status. This paper empirically studies the Event Salience task and proposes two salience detection models based on content similarities and discourse relations. The first is a feature based salience model that incorporates similarities among discourse units. The second is a neural model that captures more complex relations between discourse units. Tested on our new large-scale event salience corpus, both methods significantly outperform the strong frequency baseline, while our neural model further improves the feature based one by a large margin. Our analyses demonstrate that our neural model captures interesting connections between salience and discourse unit relations (e.g., scripts and frame structures).
Introduction
In natural language processing, extracting prominent information from text is a core challenge. While there is a growing body of work on identifying the "salience" (importance) of specific entities (people, places, organizations), the salience of events has been largely under-explored. Events form the backbone of human communication; however, not all events in a text carry the same weight. In a news article about a legal battle, the event "trial" is central to the discourse, while a passing mention of a past "war" might be irrelevant background. Traditional summarization or extraction methods often rely on simple heuristics like word frequency or how early a word appears in the text. This research aims to build models that understand semantic and narrative relationships between discourse units to accurately rank event importance.
Methodology
The problem is framed as a ranking task, and two primary models are proposed:
- Feature-Based Model (LeToR): A Learning-to-Rank model combining basic discourse features (frequency, sentence location) with content features. Content features utilize raw cosine similarities between pre-trained word embeddings of events and entities to measure an event's relatedness to the document.
- Kernel-based Centrality Estimation (KCE): This neural model moves beyond raw cosine similarity, which is often optimized for general word similarity rather than narrative discourse. KCE projects events and entities into an embedding space and uses Gaussian kernels (soft-count kernels) to weigh specific latent relations. This allows the model to capture task-relevant semantic interactions, such as narrative scripts (e.g., "arrest" → "charge") and frame-argument structures.
The Event Salience Corpus
To facilitate this research, the authors introduced the Event Salience Corpus, a massive new benchmark built from the Annotated New York Times corpus.
As shown in Table 1, the dataset contains over 500,000 training documents. Because manual labeling at this scale is prohibitive, a heuristic was used: an event in the main text is labeled as "salient" if its lemma appears in the expert-written abstract of the article. A manual validation study confirmed this heuristic serves as a strong proxy for human-judged salience.
Experimental Results
The KCE model demonstrates significant improvements over both traditional baselines and the feature-based LeToR model across multiple metrics.
KCE achieved a Precision@1 (P@01) of 50.49%, outperforming the Frequency baseline (45.42%) and the LeToR model (47.53%). In terms of Area Under the Curve (AUC), KCE reached 0.6557, a 14.41% relative improvement over the Frequency baseline. Notably, KCE showed substantial gains in Recall (R@01, R@05, R@10), indicating its effectiveness in ranking the most important events at the top of the list.
Key Insights and Discussion
- Narrative Cohesion: Analysis suggests that salient events form cohesive narrative clusters. An "Intrusion Test" revealed that salient events are more tightly connected semantically than non-salient ones, effectively forming the core "script" of a document.
- Entity Context: Ablation studies showed that removing entity information leads to a performance drop, highlighting that relationships between events and the entities involved are vital for determining importance.
- Beyond Similarity: KCE learns that specific narrative relationships (scripts and frames) dictate importance more accurately than simple embedding similarity.
Conclusion
While the current dataset uses binary labels and heuristic-based annotation, it provides a robust foundation for event salience research. Future work may explore continuous salience scoring and the use of KCE-like models for large-scale, automatic discovery of semantic relations, such as mining new narrative scripts directly from text.
Summary updated