Introduction
The cryptocurrency market is characterized by extreme volatility, often influenced more by social media sentiment and "hype" than by traditional economic indicators. Identifying and profiling key influencers on platforms like Twitter is crucial for understanding market dynamics. This paper details the approach taken by researchers at the University of Zurich for the PAN-2023 Cryptocurrency Influencer Profiling task, which requires classifying users into five levels of influence: no influencer, nano, micro, macro, and mega.
The primary challenge of this task lies in its low-resource nature. With only 32 labeled users per category and a limited number of tweets per user, standard deep learning models are prone to overfitting and struggle to capture the specific nuances of "crypto-speak" and informal Twitter structures.
Methodology: The Two Body Model
To address the data scarcity and the complexity of Twitter data, the authors proposed a "Two Body Model" ensemble architecture designed for robust few-shot learning.
Architecture and Model Selection
Instead of relying on a single transformer, the system extracts [CLS] token embeddings from the final hidden layers of two distinct models. These embeddings are concatenated and passed through a Multi-Layer Perceptron (MLP) with GELU activation for final classification. Two specific configurations were evaluated:
- Model 1 (Efficiency): A combination of DistilBERT and ELECTRA-Small.
- Model 2 (Performance): A combination of TwHIN-BERT (pre-trained on 7 billion multilingual tweets) and Twitter RoBERTa Large.
Head First Fine-tuning (HeFiT)
To preserve the extensive pre-trained knowledge within these models while adapting to a tiny dataset, the researchers employed a two-stage fine-tuning strategy called HeFiT. Initially, the transformer encoders are frozen, and only the classification head is trained for 3 epochs. Subsequently, the entire network is unfrozen and fine-tuned for an additional 17 epochs to prevent the pre-trained weights from being "washed away."
Input Strategy
The study found that concatenating a user's tweets into a single input block (up to 512 tokens) generally outperformed classifying individual tweets and using majority voting. This "Concatenation Method" allows the model to capture a broader context of a user's online persona, though ELECTRA was a notable exception, performing better with the single-tweet approach.
Key Results
The ensemble approach demonstrated a clear advantage over individual models. While the best individual model using the concatenation method (Twitter RoBERTa Large) achieved an accuracy of 53.3%, the Model 2 ensemble significantly improved upon this baseline.
As shown in the results, Model 2 (TwHIN-BERT + Twitter RoBERTa Large) achieved a peak test accuracy of 67% and a Macro F1-score of 0.63. This performance validates the hypothesis that domain-specific pre-training (on Twitter data) and ensemble methods are highly effective in few-shot learning scenarios.
Class-Level Analysis
A deeper dive into the classification metrics reveals significant variance in how the model handles different influencer tiers, highlighting the difficulty of distinguishing between adjacent levels of influence.
The model achieved perfect precision (1.00) for both the no influencer and macro categories. This indicates that when the model identifies a user as belonging to these classes, it is highly reliable. However, the recall for these classes was notably lower (0.33), suggesting that many true macro influencers and non-influencers are being misclassified into other categories like micro or mega, which showed higher recall but lower precision.
Conclusion
The University of Zurich's "Two Body Model" demonstrates that combining domain-specific language models with strategic fine-tuning can overcome the limitations of extremely small datasets in social media profiling. Future work will explore the use of prompting methods and larger generative models like T5 to further refine the distinction between subtle influencer categories and improve recall across all classes.
Summary updated