Abstract
Representing events as dense vectors is a fundamental challenge in Natural Language Processing, essential for tasks like narrative prediction and story generation. We present the SWCC (Simultaneous Weakly supervised Contrastive learning and Clustering) framework, which addresses the limitations of existing event representation methods. By treating co-occurring events as weak supervision and integrating prototype-based clustering with contrastive learning, SWCC captures nuanced semantic and relational properties—such as causality and sentiment—without the need for expensive manual annotations. Our results demonstrate that SWCC significantly outperforms state-of-the-art baselines on event similarity and narrative cloze tasks.
Introduction: The Challenge of Event Representation
In NLP, an "event" (e.g., "PersonX graduates from college") is defined not just by its lexical content, but by its complex web of relationships. Graduating is temporally linked to receiving a diploma, causally linked to career aspirations, and carries a positive sentiment.
Existing representation methods often struggle with two primary issues:
- Noisy Supervision: Relying solely on document-level co-occurrence often provides a signal that is too coarse to distinguish fine-grained relations.
- The Contrastive Conflict: Standard contrastive learning objectives (e.g., InfoNCE) push "negative" samples away from an anchor. However, in event space, a "negative" sample might be semantically similar to the anchor (e.g., "buying a car" vs. "purchasing a vehicle"). Forcing these apart degrades the quality of the learned embeddings.
The SWCC Framework
The SWCC framework mitigates these issues through a dual-pathway architecture that combines weakly supervised contrastive learning with semantic clustering.
Weakly Supervised Contrastive Learning
Unlike standard contrastive learning that uses a single positive pair, SWCC allows for multiple positives. It leverages co-occurring events from raw text as additional positive samples, weighting them by their frequency of co-occurrence. This approach allows the model to learn from the "superset" of relations implicitly present in large-scale unlabeled corpora.
Prototype-based Clustering
To resolve the contrastive conflict, SWCC introduces prototypes—representative embeddings for groups of related events. Inspired by the SwAV algorithm, the model performs online clustering while enforcing consistency: different augmented "views" of the same event must be assigned to the same prototype. This clustering acts as a semantic regularizer; even if the contrastive objective attempts to push two similar events apart, the clustering objective pulls them back together if they share a common prototype.
Joint Optimization
The model is trained using a joint loss function combining contrastive loss ($L_{cl}$), clustering loss ($L_{cp}$), and a standard Masked Language Modeling ($L_{mlm}$) loss to ensure the underlying BERT encoder retains basic linguistic knowledge.
Experimental Setup and Results
The model was trained on the New York Times Gigaword Corpus, with over 4 million event triples extracted via the Ollie system. Evaluation focused on three key areas:
- Hard Similarity: SWCC achieved 80.9% accuracy on the original task and 72.1% on an extended version, outperforming the previous best model (UniFA-S).
- Transitive Sentence Similarity: The model showed strong correlation with human-annotated similarity scores for transitive sentences.
- Multiple Choice Narrative Cloze (MCNC): In a zero-shot setting, SWCC reached 44.50% accuracy, significantly surpassing BERT-based contrastive models (39.23%).
Ablation studies confirmed that the clustering component is vital; removing it led to a significant performance drop, particularly in capturing transitive similarity.
Discussion and Future Work
A key insight of this work is that co-occurrence in raw text serves as a powerful proxy for explicit discourse relations. Furthermore, the learned prototypes were found to implicitly capture specific event types and causal chains (e.g., financial distress leading to cost-cutting) without explicit supervision.
While the number of prototypes ($M$) is a sensitive hyperparameter, performance was found to plateau around $M=10$ for the evaluated datasets. Future research will explore scaling SWCC to more diverse datasets to determine if higher prototype granularity can capture even more subtle event dynamics.
Summary updated