TL;DR
This paper tackles the challenge of classifying fine-grained real-world events in images, which is difficult due to limited labeled data. They propose a novel approach that leverages knowledge graphs (like Wikidata and Wikipedia) to create descriptive "prompts" for vision-language models (VLMs), significantly improving few-shot event classification accuracy with minimal training data.
What Problem Does This Paper Address?
Identifying specific real-world events in images is crucial for tasks like misinformation detection and news analysis. However, existing methods often rely on large, labeled datasets, which are scarce for fine-grained events (e.g., "2020 U.S. Presidential Election" vs. general "election"). This data scarcity makes traditional supervised learning approaches ineffective. While recent vision-language models (VLMs) show promise in zero-shot and few-shot learning through "prompting," their application to event classification, especially using external knowledge, remains underexplored. The paper aims to bridge this gap by developing methods to create effective prompts for VLMs, leveraging knowledge graphs to enhance event understanding and classification with limited data.
Methods
The core of the paper's contribution lies in its novel prompting strategies for VLMs, particularly CLIP. They propose four main techniques:
- Hard Prompts:
- Prompts based on Class Labels (PCL): Simple event names (e.g., "election").
- Prompts based on Static Templates (PST): Predefined sentence structures (e.g., "This is a photo of a [Class]").
- Prompts based on Wikidata Descriptions (PWD): Utilizing structured descriptions from Wikidata for each event.
- Prompts based on Wikipedia Summaries (PWS): Employing summaries from Wikipedia to provide richer context.
- Soft Prompt Learning (SPL): They adapt the CoOp method to automatically learn "soft prompts" (learnable context vectors) from a small number of training examples, allowing the model to adapt to specific events.
- Prompt Ensemble: To harness the strengths of different prompts, they combine the predictions from multiple hard and soft prompts, averaging their similarity scores to achieve a more robust classification.
- CLIP Fine-tuning: They also fine-tune the CLIP model on a large news article dataset (MMG-NewsPhoto) to improve its understanding of news-related content.
Datasets
The paper utilizes several datasets for evaluation:
- Public Benchmarks:
- Visual Event Classification Dataset (VisE-D)
- Web Image Dataset for Event Recognition (WIDER)
- Rare Events Dataset (RED)
- Social Event Dataset (SocEID)
- Newly Introduced Dataset:
- Event Instances Dataset: This novel dataset comprises 6,256 images across 184 fine-grained event instances from six broad categories (election, referendum, epidemic, protest, political campaign, natural disaster). It aims to address the lack of diverse, real-world event instances in existing benchmarks.
Key Insights and Ideas
- Knowledge Graphs are Powerful for Event Context: Simply using the event name (PCL) is insufficient. Descriptions from knowledge graphs (PWD, PWS) significantly improve zero-shot classification by providing richer context, especially for fine-grained events.
- Ensembling Prompts is Key: Combining multiple prompt types (hard and soft, including knowledge graph information) leads to superior performance compared to any single prompt type. This ensemble approach effectively leverages diverse information sources.
- Soft Prompts Need Some Data, but Ensembles are Data-Efficient: While soft prompt learning (SPL) alone requires a moderate number of training samples (around 10 per class) to outperform zero-shot methods, the prompt ensemble approach can achieve comparable or better results with as few as 2 training samples per class.
- Fine-tuning CLIP for News Helps, but Isn't Always Superior: Fine-tuning CLIP on a news-specific dataset (CLIP-MMG) shows promise, but its performance is not consistently better than the general CLIP-WIT model, especially on the Event Instances dataset. This suggests that the quality of the fine-tuning data and strategy is critical.
- Zero-Shot is Viable for Data-Scarce Scenarios: The proposed prompting and ensemble methods demonstrate that zero-shot event classification can achieve competitive results, even outperforming supervised baselines when training data is extremely limited.
Key Results
- Zero-Shot Performance: The ensemble of hard prompts (PCL, PST, PWD, PWS) achieved competitive results with zero training data, outperforming PST and PCL individually. PWD and PWS showed particular strength on fine-grained datasets like Event Instances and RED.
- Few-Shot Performance: The ensemble of soft and hard prompts (SPL, PST, PWD, PWS) significantly boosted performance in few-shot settings. This ensemble approach required only about 2 images per class to outperform other methods, whereas SPL alone needed around 10.
- Comparison to Baselines: The proposed ensemble approach achieved better or comparable results to state-of-the-art supervised baselines on most datasets, especially in few-shot scenarios with limited training data. For instance, on the Event Instances dataset, the ensemble with 30 samples per class achieved 83.38% accuracy, while a linear probe achieved 71.57%.
- Event Instances Dataset Analysis: Table 3 shows that event instance classification is challenging, with natural disasters performing best and political campaigns showing room for improvement. The ensemble approach consistently outperformed individual soft prompts.
Limitations and Future Work
- Limitations:
- The fine-tuning of CLIP (CLIP-MMG) did not always lead to superior performance, possibly due to the sampling strategy of image-text pairs not guaranteeing strong correlation or event mention.
- For some events, Wikidata descriptions were missing, requiring fallback to simpler prompts.
- The paper notes that when training data becomes more abundant (e.g., 30+ samples per class), supervised methods like linear probes can eventually outperform prompting approaches.
- Future Work:
- Investigating more advanced prompt learning techniques.
- Exploring ways to incorporate information from longer descriptions (e.g., full Wikipedia articles).
- Integrating ontology information from knowledge graphs more deeply into prompts.
Summary updated