TL;DR
This paper presents the winning system for the ProfNER shared task, which aims to identify professions and occupations in Spanish tweets. The authors achieved top performance by fine-tuning a pre-trained transformer model (BETO) and solving both classification and Named Entity Recognition (NER) tasks simultaneously, demonstrating the power of transfer learning for specialized NLP tasks.
What Problem Does This Paper Address?
The paper tackles the challenge of identifying professions and occupations within social media text, specifically Spanish tweets, as part of the SMM4H (Social Media Mining for Health Applications) Shared Task 2021. This task, known as ProfNER, is crucial for understanding health-related issues and epidemiological trends by characterizing the roles and statuses of individuals mentioned online. The task is split into two sub-tasks: (7a) binary classification to determine if a tweet mentions an occupation, and (7b) Named Entity Recognition (NER) to pinpoint and categorize occupation mentions as either "PROFESION" (profession) or "SITUACION_LABORAL" (working status).
Methods
The authors developed two systems, with the first system being the winning entry. Both systems share a common architecture: a backbone model for feature extraction and contextualization, and a task head for specific operations and loss computation.
The winning system (System 1) utilizes a transformer-based pretrained language model, specifically BETO (a BERT model trained on a large Spanish corpus), distributed via Hugging Face. The input word tokens are further tokenized into word pieces using the BERT tokenizer, with special tokens like [CLS] and [SEP] added. A novel approach to handle problematic word tokens that the tokenizer ignores (like newline characters) is to replace them with a dummy token "æ" to ensure correct alignment. The output from the transformer is then fed into the task head.
The task head is designed to solve both sub-tasks (7a and 7b) simultaneously. For task 7a, a tanh activation layer followed by a linear classification layer is used, with the loss calculated via Cross Entropy. For task 7b, to bridge the gap between word piece features and word token predictions, a subword pooling strategy is employed where the first word piece represents the entire word token. This is followed by a linear classification layer and a Conditional Random Field (CRF) model for NER tag prediction.
System 2, designed for efficiency, uses a Recurrent Neural Network (RNN) backbone. This system combines pre-trained skip-gram word vectors with character-level embeddings processed by a bidirectional GRU. These are then fed into another larger bidirectional GRU for contextualization. The task head for System 2 uses a bidirectional LSTM for pooling and a classification layer for task 7a, and a feedforward network with a linear classification layer followed by a CRF for task 7b.
Both systems were trained using biome.text, a library built on AllenNLP and PyTorch. Hyperparameter optimization (HPO) was performed using Ray Tune, employing a tree-structured Parzen Estimator and ASHA trial scheduler, with the F1 score of task 7b as the primary metric.
Datasets
The primary dataset used for training and evaluation is the ProfNER dataset, which consists of Spanish tweets annotated for occupation mentions. The authors mention that no data augmentation or external data was used for training. The dataset was split into training and validation sets for hyperparameter tuning, and the final evaluation was performed on a combined training and validation set for test results.
Key Insights and Ideas
- Simultaneous Task Solving is Beneficial: The authors observed a clear correlation between the classification F1 score (task 7a) and the NER F1 score (task 7b), suggesting that solving both tasks concurrently provides beneficial feedback and leads to better overall performance.
- Transformer Dominance: The transformer-based system (System 1) significantly outperformed the RNN-based system (System 2), even with extensive architecture searches and hyperparameter tuning for the RNN. This highlights the superior transfer learning capabilities of pre-trained transformers for specialized NLP tasks, especially when training data is scarce.
- Importance of Pre-trained Models: The success of System 1 underscores the value of leveraging large, pre-trained language models like BETO. Fine-tuning these models on domain-specific data can yield highly competitive results with relatively straightforward architectures.
- Handling Tokenization Challenges: The simple yet effective method of replacing ignored word tokens with a dummy token ("æ") demonstrates a practical approach to ensure data integrity and alignment during the complex tokenization process of transformer models.
Key Results
System 1, the transformer-based winning system, achieved the highest scores on both ProfNER sub-tasks:
- Task 7a (Classification): F1 Test: 0.93, F1 Valid: 0.92
- Task 7b (NER): F1 Test: 0.839, F1 Valid: 0.834
These results were, on average, 8 points above the median of all submissions. System 1 had a model size of approximately 1.1 x 10⁸ parameters and an inference time of 24.5 ms per prediction.
System 2, the RNN-based system, also performed above the arithmetic median for both tasks (F1 Test: 0.88 for 7a, 0.764 for 7b) but was significantly outperformed by System 1. System 2 was considerably smaller (1.5 x 10⁷ parameters) and faster (3.7 ms per prediction). The authors noted that adding validation data to the training set for System 2 led to a significant performance boost, while System 1's performance remained largely unchanged, suggesting System 2 was more sensitive to the amount of training data.
Limitations and Future Work
- Gazetteer Inclusion: The authors acknowledge that they did not utilize the provided gazetteers (lists of known occupations) during the ProfNER track. They believe incorporating these could have further improved precision, which was consistently lower than recall for both tasks.
- Data Scarcity: The paper implicitly highlights the challenge of limited training data for specialized tasks, where pre-trained models offer a significant advantage.
- Future Work: The authors suggest that exploring the inclusion of gazetteers is a promising avenue for future work. They also conclude that leveraging transfer learning from pre-trained models and fine-tuning them for the target domain is a conceptually simple and effective approach for achieving competitive performance, especially in low-resource scenarios.
Summary updated