Abstract
Representations of events described in text are important for various tasks. In this work, we present SWCC: a Simultaneous Weakly supervised Contrastive learning and Clustering framework for event representation learning. SWCC learns event representations by making better use of co-occurrence information of events. Specifically, we introduce a weakly supervised contrastive learning method that allows us to consider multiple positives and multiple negatives, and a prototype-based clustering method that avoids semantically related events being pulled apart. For model training, SWCC learns representations by simultaneously performing weakly supervised contrastive learning and prototype-based clustering. Experimental results show that SWCC outperforms other baselines on Hard Similarity and Transitive Sentence Similarity tasks. In addition, a thorough analysis of the prototype-based clustering method demonstrates that the learned prototype vectors are able to implicitly capture various relations between events.
Introduction
In Natural Language Processing, representing "events" (e.g., “PersonX applies for a job”) in a machine-readable vector space is a fundamental challenge. High-quality event representations are crucial for downstream applications such as story generation, script prediction, and discourse analysis. However, events are inherently complex, encompassing sentiments, temporal sequences, and causal links.
Existing approaches to event representation generally fall into two categories. Co-occurrence-based methods assume that events appearing in the same document are related. While this provides access to vast amounts of "cheap" data, these methods often use the signal too simplistically, typically focusing on single positive/negative pairs. Conversely, knowledge-based methods utilize manually labeled discourse relations or commonsense knowledge. While high in quality, these labels are expensive to scale and cannot easily account for the near-infinite variety of event types.
We propose that document-level co-occurrence is actually a "superset" of many semantic relations. If two events appear together, they likely share a topic, a causal link, or a temporal sequence. The challenge lies in extracting this rich information effectively without human intervention.
The SWCC Framework
To address these challenges, we introduce SWCC (Simultaneous Weakly supervised Contrastive learning and Clustering). Built upon a BERT backbone, the framework employs three integrated components to refine event embeddings.
1. Weakly Supervised Contrastive Learning
Unlike standard contrastive learning, which typically uses a single positive (such as a data augmentation of the anchor), SWCC utilizes multiple positives. These include different dropout-augmented versions of the anchor event and, crucially, events that co-occur with it in the same document. To mitigate noise, co-occurring events are weighted by their frequency in the corpus; the more frequently two events appear together globally, the more strongly they are pulled together in the embedding space.
2. Prototype-based Clustering
A common pitfall of contrastive learning is that it may inadvertently push semantically related events apart if they are not explicitly labeled as positives. SWCC addresses this by maintaining a set of "prototypes"—representative embeddings that act as semantic anchors for clusters. Using the Sinkhorn-Knopp algorithm, the model assigns events to these prototypes to ensure an even distribution and prevent "representation collapse" (where all events map to a single point). The model is trained to ensure that different augmentations of the same event result in the same cluster assignment, keeping related concepts like "earning fees" and "banks" in the same neighborhood.
3. Joint Training Objective
The final training objective is a multi-task loss function: $$L = L_{cl} + L_{cp} + L_{mlm}$$ This combines the contrastive loss ($L_{cl}$), the clustering loss ($L_{cp}$), and a standard Masked Language Modeling ($L_{mlm}$) loss. This ensures the model learns specific event relations while retaining the foundational linguistic knowledge inherent in pre-trained language models.
Experimental Evaluation
We trained SWCC on 4,029,877 distinct event triples extracted from the New York Times Gigaword Corpus using the Ollie open information extraction system. The model was evaluated across several rigorous benchmarks:
- Hard Similarity Task: A dataset of event pairs designed to be deceptive, featuring high lexical overlap with different meanings, or low overlap with similar meanings. SWCC achieved 80.9% accuracy on the original set and 72.1% on the extended set, significantly outperforming previous models like UniFA-S and NTN-IntSent.
- Transitive Sentence Similarity: On a dataset of 108 human-annotated transitive sentence pairs, SWCC reached a Spearman’s correlation of 0.82, a substantial improvement over the previous state-of-the-art of 0.75.
- Multiple Choice Narrative Cloze (MCNC): In a zero-shot transfer setting, the model predicted the next event in a chain with 44.50% accuracy, demonstrating that the learned representations generalize well to narrative reasoning.
Discussion and Insights
The success of SWCC highlights the value of treating co-occurrence as a broad umbrella for discourse markers. By weighting these signals, the model implicitly captures causal and temporal relations without explicit labels. Furthermore, the learned prototypes naturally organize into thematic clusters (e.g., financial, political, or social events), providing a structured semantic space.
However, the framework is not without limitations. The model's performance is partially dependent on the quality of the initial event extraction; errors in the Ollie system (such as misidentified subjects) can propagate into the embeddings. Additionally, the number of prototypes ($M$) is currently a fixed hyperparameter. Future work may explore dynamic or hierarchical clustering to better accommodate the diversity of massive datasets, or integrate structural information from knowledge graphs to further refine prototype assignments.
Summary updated