Word Embeddings, Cosine Similarity and Deep Learning for Identification of Professions & Occupations in Health-related Social Media

Sergio Santamaría Carrasco, Roberto Cuervo RosilloNAACL 2021

TL;DR

This paper presents a deep learning system for identifying professions and occupations in Spanish health-related tweets, achieving a strong F1 score of 0.823. The approach combines various word embeddings, including pre-trained Spanish BERT, with an encoder-decoder architecture and cosine similarity to capture nuanced contextual information.

What Problem Does This Paper Address?

In critical situations like pandemics, understanding which occupations are most vulnerable (due to direct exposure or mental health impacts) is crucial for implementing effective preventive measures. Social media, particularly Twitter, has become a rich source of real-time information. However, extracting this specific information requires sophisticated Natural Language Processing (NLP) techniques. This paper tackles the challenge of accurately recognizing professions and occupations mentioned in Spanish tweets within the health domain, a task that is vital for public health monitoring and response.

Methods

The core of the proposed system is an encoder-decoder architecture with an attention mechanism, a setup previously successful for temporal expression recognition. This architecture is augmented with a rich set of features:

  • Word Embeddings: The system utilizes multiple word embeddings. This includes pre-trained Spanish medical corpus embeddings and Spanish Twitter data embeddings related to COVID-19 (both 300-dimensional FastText). Crucially, it also incorporates contextual embeddings from a fine-tuned Spanish BERT model (BETO), which dynamically adapt to surrounding words.
  • Cosine Similarity: Embeddings of entities from the training and validation sets are used to compute cosine similarity. This technique, shown to improve Twitter data analysis, generates a 3717-dimensional vector.
  • Part-of-Speech (PoS) Tagging: A 40-dimensional embedding representation of PoS tags is learned, aiding in word sense disambiguation and capturing local word context.
  • Character and Syllable Embeddings: Character-level embeddings (30-dimensional) and syllable-level embeddings (75-dimensional) are learned. These are particularly useful for handling out-of-vocabulary words and capturing morphological nuances in Spanish.
  • Architecture Details: These diverse features are fed into a convolutional and global max pooling block before being processed by the encoder-decoder architecture. A fully connected dense layer with a tanh activation function and a Conditional Random Fields (CRF) layer for label decoding complete the pipeline.

Datasets

The system was trained and evaluated using data from the ProfNER-ST (Miranda-Escalada et al., 2021b) challenge, specifically the 7b subtrack of the sixth Social Media Mining for Health Applications (SMM4HA) (Magge et al., 2021). The training set used was the ProfNER corpus (Miranda-Escalada et al., 2020), which provides gold-standard annotations for profession detection in Spanish COVID-19 tweets. The development set was used for hyperparameter fine-tuning, and both sets were combined for the final prediction stage.

Key Insights and Ideas

  • The Power of Diverse Embeddings: Combining multiple types of word embeddings (static, contextual, domain-specific) significantly boosts performance. The inclusion of Spanish BERT (BETO) embeddings, which capture context, was particularly impactful.
  • Cosine Similarity for Context: Leveraging cosine similarity on BETO embeddings of entities proved beneficial, suggesting that capturing semantic similarity between words in a health context is valuable.
  • Morphological Richness Matters: For morphologically rich languages like Spanish, character and syllable-level embeddings are not just supplementary but essential for handling variations and out-of-vocabulary terms.
  • Encoder-Decoder with Attention is Robust: The chosen encoder-decoder architecture with attention, a proven technique, effectively integrates these diverse features for sequence labeling.

Key Results

The proposed system achieved a micro-averaged F1-score of 0.823 on the official test set of the ProfNER-ST challenge. This indicates a high level of accuracy in identifying professions and occupations within Spanish health-related tweets. The validation set performance was also strong, with a Precision of 0.893, Recall of 0.753, and F1-Score of 0.817.

Limitations and Future Work

The authors acknowledge that future work could explore different Data Augmentation techniques to further enhance the model's robustness. Additionally, they suggest expanding the scope to include other entities like companies or organizations, which could provide further valuable context for identifying occupations.

Summary updated

BibTeX
@inproceedings{carrasco2021word,
  author = {Sergio Santamaría Carrasco and Roberto Cuervo Rosillo},
  title = {{Word Embeddings, Cosine Similarity and Deep Learning for Identification of Professions \& Occupations in Health-related Social Media}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure