TL;DR
This paper tackles the challenge of automatically detecting alcohol advertisements in digital media, a crucial step in protecting adolescents from harmful marketing. They introduce a novel genetic algorithm approach to optimize prompts for Multimodal Large Language Models (MLLMs), achieving high accuracy in identifying these ads.
What Problem Does This Paper Address?
Adolescents are increasingly exposed to advertisements for addictive substances, particularly through online platforms like social media. This exposure is linked to higher rates of substance use. While traditional media has been studied, the dynamic and often user-generated nature of online content makes detection difficult. Existing methods often focus on specific image elements or require extensive manual prompt engineering. This paper aims to automate the optimization of prompts for MLLMs to effectively identify alcohol advertisements within digital media, even when they are not the sole focus of an image.
Methods
The core of this research is an adaptive prompt engineering approach using a genetic algorithm. Here's how it works:
- MLLM as the Detector: They use Multimodal Large Language Models (MLLMs) that can process both images and text to identify alcohol ads.
- Genetic Algorithm for Prompt Optimization: Instead of manually crafting prompts, they use a genetic algorithm to evolve them.
- Individuals: Each "individual" in the algorithm is a combination of an instruction prompt (telling the MLLM what to look for) and a mutation prompt (used to generate the instruction prompt).
- Fitness: The "fitness" of an individual is determined by the accuracy of the MLLM's performance when using its instruction prompt to classify a batch of images.
- Mutation: Prompts are mutated using techniques inspired by established Large Language Model (LLM) prompt engineering strategies like Chain-of-Thought (CoT), Generated Knowledge, Self-critique, and Expert Prompting.
- Selection: A roulette wheel selection method is used to ensure diversity and prevent the algorithm from getting stuck in local optima, favoring prompts with higher fitness but still allowing lower-fitness prompts a chance to evolve.
- Prompt Engineering Techniques: The mutation process leverages techniques like:
- Chain-of-Thought (CoT): Encouraging step-by-step reasoning.
- Generated Knowledge: Providing additional relevant information.
- Self-critique: Having the model evaluate its own output.
- Expert Prompting: Instructing the model to act as an expert in prompt engineering.
- Output Classification: The MLLM's text output is then classified by a BERT text classifier to determine if an alcohol ad is present.
Datasets
- RICO Dataset: The authors augmented the RICO dataset, which consists of Android user interface (UI) screenshots. They superimposed alcohol advertisements onto 2,100 of these screenshots, creating a dataset of 4,200 images (2,100 original UI screenshots and 2,100 with superimposed ads). This augmented dataset was split into training (3,200 images) and testing (1,000 images) sets.
- Alcohol Ad Images: Additional alcohol advertisement images were collected from Google searches.
Key Insights and Ideas
- Automated Prompt Optimization is Key: The performance of MLLMs is highly dependent on prompt quality, and automating this process with a genetic algorithm is a powerful way to find optimal prompts.
- Diversity in Mutation is Crucial: Using a roulette wheel selection method, rather than a purely greedy approach, is vital for preventing the genetic algorithm from converging to a local maximum and for exploring a wider range of effective prompts.
- MLLMs Can Detect Ads Beyond the Obvious: The approach is designed to identify ads even when they are not the sole focus of an image, which is a significant advantage over methods that analyze entire images in isolation.
- Open-Source MLLMs are Viable: The performance of the open-source LLaVA model was comparable to the proprietary GPT-4 model, suggesting that accessible tools can be developed for this task. This is particularly important for privacy and cost-effectiveness.
- Prompt Engineering Techniques Transfer: Prompt engineering techniques developed for LLMs can be effectively adapted and utilized within an MLLM context.
Key Results
- The optimized prompts achieved an accuracy of 0.94 and an F1-score of 0.94 when using the GPT-4 vision model.
- The classifier employing the LLaVA model achieved an accuracy of 0.922 and an F1-score of 0.9215.
- The Chain-of-Thought (CoT) mutation type was consistently the most effective throughout the generations for both GPT-4 and LLaVA models.
- The BERT classifier achieved an accuracy of 0.98.
Limitations and Future Work
- Bias: The authors acknowledge that the optimization strategy does not explicitly address potential biases in the training data, which could lead to biased prompts. They suggest that generative models can exhibit biases, requiring careful examination.
- Resource Constraints: Limitations in computational resources (using a paid MLLM and a 7 billion parameter LLaVA model) restricted the scope of experiments and the exploration of hyperparameters.
- Language Specificity: The collected alcohol ads were exclusively in English, limiting the model's multilingual capabilities.
- Ethical Considerations: The technology, while beneficial for public health, carries risks of misuse for surveillance or accessing sensitive information. The authors emphasize the need for a balanced ethical framework, robust privacy safeguards, and bias mitigation.
- Future Work: The authors envision extending the algorithm to detect other harmful substances (tobacco, drugs) and adapting it for online platforms to foster a safer online environment. They also suggest exploring larger LLaVA models and conducting more extensive hyperparameter tuning.
Summary updated