TL;DR
This paper investigates whether large language models (LLMs) like GPT-4 and LLaMA 2 outperform traditional machine learning pipelines on text classification tasks. Surprisingly, a pre-LLM era ensemble model (Electra + XGBoost) achieved better performance on news trustworthiness classification, despite being orders of magnitude smaller. This highlights that LLMs are not a universal solution and traditional methods still hold significant value.
What Problem Does This Paper Address?
Large Language Models (LLMs) have exploded in popularity, with applications extending far beyond simple text generation. One area of significant interest is text classification, where LLMs offer the allure of zero- or few-shot learning, potentially eliminating the need for extensive training data. However, the rapid adoption of LLMs has outpaced our understanding of their limitations, including robustness issues, biases, and the tendency to "hallucinate" (generate factually incorrect information). This paper aims to provide a concrete case study comparing the performance, training, and deployment requirements of popular LLMs against a well-established pre-LLM era classification pipeline on a specific, challenging task: news trustworthiness classification. The goal is to offer a template for evaluating LLMs against older methods, moving beyond just accuracy to consider practical aspects.
Methods
The study compares two distinct approaches to text classification:
- Modular Ensemble Pipeline (Pre-LLM Era): This approach, exemplified by "Electra + XGBoost," involves a multi-stage process. First, several "feature models" (fine-tuned Electra models) are trained on specific aspects of the text (e.g., identifying clickbait, hate speech, political bias). These models extract feature representations from the input article. Second, a meta-model (XGBoost classifier) takes these feature representations as input to make the final classification. This method requires training for each feature model and the meta-model.
- LLM Pipeline (LLM Era): This approach leverages the emergent capabilities of LLMs. The entire classification task is delegated to a single LLM (GPT-4 or LLaMA 2). Information about the task, including the methodology and desired output format, is conveyed through a "system prompt" in natural language. The LLMs are used in a zero-shot manner, meaning they are not fine-tuned on the specific dataset.
Datasets
The primary dataset used for evaluation is the Verifee news trustworthiness dataset. This dataset contains over 10,000 Czech news articles and is designed for a two-stage classification problem that requires the model to provide reasoning for its predictions. The dataset defines four classes of credibility: credible, partially credible, partially manipulative, and manipulative.
For the modular ensemble pipeline, several smaller datasets were used to fine-tune the individual feature models. These include:
- GoEmotions (for anger detection)
- Kaggle Clickbait Dataset (for clickbait detection)
- HateXplain (for hate speech detection)
- German News Bias Dataset (for political bias detection)
- StereoSet (for stereotypization detection)
- Kaggle News Category Dataset (for seriousness/tabloid news detection)
Key Insights and Ideas
- LLMs are not a silver bullet: The most striking finding is that a traditional, smaller model (Electra + XGBoost) outperformed powerful LLMs (GPT-4, LLaMA 2) on the news trustworthiness task. This challenges the assumption that LLMs are universally superior for all NLP tasks.
- Size and complexity don't always equal performance: The pre-LLM ensemble pipeline is orders of magnitude smaller in parameter size and computational requirements compared to the LLMs, yet it achieved better results.
- Task-specific expertise matters: The modular approach, by breaking down the problem into specialized feature extraction, seems to have captured nuances of news trustworthiness more effectively than the general-purpose LLMs in this specific case.
- Deployment costs are a major differentiator: LLMs, despite requiring less training effort (zero-shot), demand vastly more computational resources for deployment (e.g., 3370 GB of virtual memory for GPT-4 vs. 0.9 GB for Electra + XGBoost).
Key Results
- Performance: The Electra + XGBoost pipeline achieved the highest Micro F-1 score of 0.533 on the Verifee dataset. GPT-4 achieved an F-1 score of 0.531 (Czech) and 0.425 (English), while LLaMA 2 70B achieved 0.188 (Czech) and 0.256 (English). LLaMA 2 performed particularly poorly, often predicting only one class.
- Confusion Matrices: All models struggled with the "partially credible" and "partially manipulative" classes, performing best on the extreme "credible" and "manipulative" classes.
- Training Requirements: The LLM pipelines required no additional fine-tuning, whereas the modular ensemble pipeline involved a multi-stage training process.
- Deployment Requirements: The Electra + XGBoost pipeline requires only 0.9 GB of virtual memory, while LLAMA 2 requires approximately 140 GB, and GPT-4 requires a staggering 3370 GB.
Limitations and Future Work
The authors acknowledge several limitations:
- Single Task and Languages: The comparison is limited to one classification task (news trustworthiness) and two languages (Czech and English). Performance might differ significantly on other tasks or languages.
- Default Settings: Both LLM and ensemble pipelines were evaluated in default settings. Further improvements could be achieved through hyperparameter optimization for the ensemble or fine-tuning for the LLMs, which were not explored to maintain a fair comparison of their "out-of-the-box" capabilities.
- GPT-4 Parameter Estimate: The parameter count for GPT-4 is an estimate based on external sources and is acknowledged as speculative.
The authors suggest that this case study serves as a template for similar task- and dataset-specific evaluations. They encourage more work comparing LLMs to older methods across various tasks, languages, and datasets to build a more comprehensive understanding of LLM capabilities and limitations. They also highlight the critical need for more research into the robustness of LLM-based methods.
Summary updated