Improving Event Representation via Simultaneous Weakly Supervised Contrastive Learning and Clustering

Jun Gao, Wei Wang, Changlong Yu, Huan Zhao, Wilfred Ng, Ruifeng Xu

Abstract

Representing events as dense vectors is a fundamental challenge in Natural Language Processing, essential for tasks like narrative prediction and story generation. We present the SWCC (Simultaneous Weakly supervised Contrastive learning and Clustering) framework, which addresses the limitations of existing event representation methods. By treating co-occurring events as weak supervision and integrating prototype-based clustering with contrastive learning, SWCC captures nuanced semantic and relational properties—such as causality and sentiment—without the need for expensive manual annotations. Our results demonstrate that SWCC significantly outperforms state-of-the-art baselines on event similarity and narrative cloze tasks.

Introduction: The Challenge of Event Representation

In NLP, an "event" (e.g., "PersonX graduates from college") is defined not just by its lexical content, but by its complex web of relationships. Graduating is temporally linked to receiving a diploma, causally linked to career aspirations, and carries a positive sentiment.

Conceptual overview of event representation and relational grouping.

Existing representation methods often struggle with two primary issues:

  1. Noisy Supervision: Relying solely on document-level co-occurrence often provides a signal that is too coarse to distinguish fine-grained relations.
  2. The Contrastive Conflict: Standard contrastive learning objectives (e.g., InfoNCE) push "negative" samples away from an anchor. However, in event space, a "negative" sample might be semantically similar to the anchor (e.g., "buying a car" vs. "purchasing a vehicle"). Forcing these apart degrades the quality of the learned embeddings.

The SWCC Framework

The SWCC framework mitigates these issues through a dual-pathway architecture that combines weakly supervised contrastive learning with semantic clustering.

The dual-pathway architecture of SWCC, combining contrastive learning and prototype-based clustering.

Weakly Supervised Contrastive Learning

Unlike standard contrastive learning that uses a single positive pair, SWCC allows for multiple positives. It leverages co-occurring events from raw text as additional positive samples, weighting them by their frequency of co-occurrence. This approach allows the model to learn from the "superset" of relations implicitly present in large-scale unlabeled corpora.

Prototype-based Clustering

To resolve the contrastive conflict, SWCC introduces prototypes—representative embeddings for groups of related events. Inspired by the SwAV algorithm, the model performs online clustering while enforcing consistency: different augmented "views" of the same event must be assigned to the same prototype. This clustering acts as a semantic regularizer; even if the contrastive objective attempts to push two similar events apart, the clustering objective pulls them back together if they share a common prototype.

Joint Optimization

The model is trained using a joint loss function combining contrastive loss ($L_{cl}$), clustering loss ($L_{cp}$), and a standard Masked Language Modeling ($L_{mlm}$) loss to ensure the underlying BERT encoder retains basic linguistic knowledge.

Experimental Setup and Results

The model was trained on the New York Times Gigaword Corpus, with over 4 million event triples extracted via the Ollie system. Evaluation focused on three key areas:

  1. Hard Similarity: SWCC achieved 80.9% accuracy on the original task and 72.1% on an extended version, outperforming the previous best model (UniFA-S).
  2. Transitive Sentence Similarity: The model showed strong correlation with human-annotated similarity scores for transitive sentences.
  3. Multiple Choice Narrative Cloze (MCNC): In a zero-shot setting, SWCC reached 44.50% accuracy, significantly surpassing BERT-based contrastive models (39.23%).

Ablation studies confirmed that the clustering component is vital; removing it led to a significant performance drop, particularly in capturing transitive similarity.

Discussion and Future Work

A key insight of this work is that co-occurrence in raw text serves as a powerful proxy for explicit discourse relations. Furthermore, the learned prototypes were found to implicitly capture specific event types and causal chains (e.g., financial distress leading to cost-cutting) without explicit supervision.

While the number of prototypes ($M$) is a sensitive hyperparameter, performance was found to plateau around $M=10$ for the evaluated datasets. Future research will explore scaling SWCC to more diverse datasets to determine if higher prototype granularity can capture even more subtle event dynamics.

Summary updated

Abstract

Representations of events described in text are important for various tasks. In this work, we present SWCC: a Simultaneous Weakly supervised Contrastive learning and Clustering framework for event representation learning. SWCC learns event representations by making better use of co-occurrence information of events. Specifically, we introduce a weakly supervised contrastive learning method that allows us to consider multiple positives and multiple negatives, and a prototype-based clustering method that avoids semantically related events being pulled apart. For model training, SWCC learns representations by simultaneously performing weakly supervised contrastive learning and prototype-based clustering. Experimental results show that SWCC outperforms other baselines on Hard Similarity and Transitive Sentence Similarity tasks. In addition, a thorough analysis of the prototype-based clustering method demonstrates that the learned prototype vectors are able to implicitly capture various relations between events.
BibTeX
@inproceedings{gao2022improving,
  author = {Jun Gao and Wei Wang and Changlong Yu and Huan Zhao and Wilfred Ng and Ruifeng Xu},
  title = {{Improving Event Representation via Simultaneous Weakly Supervised Contrastive Learning and Clustering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=QPd_i92YGr_},
}
Figure