NU at WASSA 2024 Empathy and Personality Shared Task: Enhancing Personality Predictions with Knowledge Graphs; A Graphical Neural Network and LightGBM Ensemble Approach

Emmanuel Osei-Brefo, Huizhi LiangWassa 2024 · DOI

TL;DR

This paper introduces a novel ensemble approach for personality prediction, combining BERT embeddings with knowledge graph features processed by Graph Neural Networks (GNNs) and LightGBM. This method significantly improves accuracy by capturing both semantic and relational information, achieving a 2nd place ranking in the WASSA 2024 Shared Task 4.

What Problem Does This Paper Address?

Predicting personality traits from text is a challenging task. Traditional machine learning models often struggle to capture the nuanced relational information present in human language, which is crucial for understanding personality. While BERT has been effective at extracting semantic meaning from text, it doesn't inherently capture the complex relationships between entities that can be found in essays. This paper aims to bridge this gap by integrating rich semantic information from BERT with structured relational data from knowledge graphs, enhanced by the power of GNNs and a gradient boosting model.

Methods

The core of this work lies in a multi-faceted approach:

  • BERT Embeddings: Pre-trained BERT models are used to generate rich contextual embeddings from user essays, capturing the semantic nuances of the text.
  • Knowledge Graph Construction and Embedding: For each user, a knowledge graph is built by extracting entities and relationships from their essays. These graphs are then embedded using Node2Vec to capture structural and relational information.
  • Graph Neural Networks (GNNs): Graph Convolutional Networks (GCNs) and Graph Attention Networks (GATs) are employed to process the knowledge graph embeddings. GCNs capture local neighborhood structures, while GATs introduce attention mechanisms to weigh the importance of different neighboring nodes.
  • LightGBM: This gradient boosting framework is used to complement the GNNs, excelling at handling large-scale data and providing efficient training.
  • Ensemble Approach: The predictions from GCN, GAT, and LightGBM are combined using a weighted average to leverage the strengths of each model.
  • Novel Loss Function: A specialized loss function is introduced, combining Mean Squared Error (MSE) loss for minimizing prediction error, Pearson correlation loss for ensuring strong correlation between predictions and targets, and contrastive loss for emphasizing relationships between example pairs.

Datasets

The study utilized the Track 4 dataset from WASSA 2024, which was merged with the Task 3 dataset to create a comprehensive dataset. This enriched dataset included:

  • Detailed personal essays written by participants.
  • Demographic information (gender, education, race, age, income).
  • Scores from the Interpersonal Reactivity Index (IRI).
  • Target scores for the Big Five personality traits (OCEAN).

Key Insights and Ideas

  • Synergy of Semantic and Relational Data: The paper strongly demonstrates that combining BERT's semantic understanding with knowledge graph's relational structure leads to significantly improved personality prediction.
  • GNNs for Relational Understanding: GNNs, particularly GCNs and GATs, are highly effective at processing the structured data from knowledge graphs, unlocking valuable insights that might be missed by purely text-based models.
  • Ensemble Power: The ensemble approach, combining GNNs and LightGBM, proves more robust than individual models, highlighting the benefit of aggregating diverse predictive strengths.
  • Importance of a Multi-faceted Loss Function: The custom loss function, balancing MSE, Pearson, and contrastive losses, is crucial for optimizing different aspects of the prediction task, from error minimization to relationship learning.

Key Results

The proposed ensemble model achieved superior performance compared to baseline models. The system officially ranked 2nd at the WASSA 2024 Shared Track 4: Personality Prediction (PER) track. Table 2 shows impressive Pearson correlation coefficients for the ensemble model on the validation set, with values like 0.482 for Openness, 0.579 for Conscientiousness, and 0.600 for Neuroticism. Table 3 further details these strong results on the test dataset, with the "Ensemble(b)" model (which incorporated hyper-parameter tuning) showing particularly high correlations, such as 0.302 for Openness, 0.380 for Agreeableness, and 0.457 for Neuroticism.

Limitations and Future Work

The authors acknowledge that extracting meaningful features using BERT and constructing knowledge graphs are computationally intensive. The quality of extracted features can also be dependent on preprocessing and entity extraction methods.

Future work will focus on refining these methods by incorporating additional data sources to further enhance performance.

Summary updated

BibTeX
@inproceedings{oseibrefo2024nu,
  author = {Emmanuel Osei-Brefo and Huizhi Liang},
  title = {{NU at WASSA 2024 Empathy and Personality Shared Task: Enhancing Personality Predictions with Knowledge Graphs; A Graphical Neural Network and LightGBM Ensemble Approach}},
  booktitle = {Wassa},
  year = {2024},
  doi = {10.18653/v1/2024.wassa-1.36},
}
Figure