TL;DR
This paper explores how Large Language Models (LLMs) can be used to analyze climate change advertisements on social media, uncovering targeting strategies and potential biases. The study demonstrates LLMs' ability to accurately predict demographic targets and provide explanations, offering a transparent way to understand microtargeting practices and identify fairness concerns.
What Problem Does This Paper Address?
Climate change communication increasingly relies on microtargeting on social media to reach specific demographic groups. However, the exact strategies and the fairness of these practices remain opaque, especially since platforms often keep their targeting logic private. This lack of transparency makes it difficult for researchers, policymakers, and the public to understand how these messages are crafted and whether they perpetuate biases. This paper aims to bridge this gap by using LLMs as a tool to perform a "post-hoc" analysis of these microtargeted ads, revealing thematic insights and evaluating fairness.
Methods
The core of this research involves using a state-of-the-art LLM (OpenAI's o1-preview model) to analyze a dataset of climate change advertisements. The LLM was tasked with two primary functions:
- Demographic Prediction with Explanation: For each ad, the LLM was asked to predict the targeted demographic (gender and age group) and provide a brief explanation for its reasoning. This "chain-of-thought" approach allows for transparency in the LLM's decision-making process.
- Thematic Analysis of Explanations: The explanations generated by the LLM were then analyzed to identify recurring themes and aspects used in targeting different demographic segments. This involved a meticulous evaluation by NLP and CSS experts to ensure accuracy and alignment with the ad content.
- Fairness Evaluation: The study employed established fairness metrics (Demographic Parity, Equal Opportunity, and Predictive Equality) to assess disparities in the LLM's prediction accuracy and error rates across different demographic groups. This helps to uncover potential biases in the LLM's performance.
Datasets
The study utilized a dataset of 21,372 climate-related English ads on Meta (Facebook) from the US, originally released by Islam et al. (2023b) and further detailed in Islam and Goldwasser (2024a). This dataset includes ad ID, title, description, spend, impressions, and distribution of impressions broken down by gender and age. For this study, the dataset was filtered to include only ads viewed exclusively by either males or females, or by specific age groups, resulting in a final dataset of 227 ads.
Key Insights and Ideas
- LLMs as Transparent Auditors: The research demonstrates that LLMs can act as powerful, "third-party" auditors for microtargeted ads, offering a window into opaque platform logic.
- Demographic Targeting Strategies Revealed: The LLM's explanations uncovered distinct thematic strategies for different demographics. For instance, young adults were often targeted with messages emphasizing activism and environmental consciousness, while women were engaged through themes of caregiving and social advocacy.
- The Nuance of Explanations: The LLM's ability to provide explanations is crucial, as it moves beyond simple classification to offer insights into why an ad might be targeted to a specific group, revealing underlying assumptions and messaging patterns.
- Fairness is Not Automatic: While LLMs perform well overall, the study highlights that biases can still exist, particularly in the classification of certain demographic groups, underscoring the need for careful fairness evaluations.
Key Results
- Overall Accuracy: The LLM achieved an overall accuracy of 88.55% in predicting targeted demographics.
- Gender Prediction: High accuracy was observed for gender prediction, with 94.92% for female-targeted ads and 85.10% for male-targeted ads.
- Age Group Prediction: Accuracy varied across age groups, with strong performance for young adults (88.00%) and early working individuals (91.46%). However, performance dropped for the late working group (75.00%) and significantly for seniors (33.33%).
- Fairness Disparities:
- Gender: A slight imbalance favored females in positive predictions, and the True Positive Rate (TPR) was higher for females (0.95) than males (0.85), though the statistical significance test for this difference was borderline.
- Age: The late working group showed a higher likelihood of positive predictions, while seniors had a significantly lower Demographic Parity ratio (0.33), indicating underrepresentation. The TPR for seniors was notably lower (0.33) compared to other groups. A statistical test confirmed a significant disparity in TPR between young adults and seniors.
- Thematic Insights:
- Male Targeting: Themes included practical and economic appeals, technology, property, traditional male activities, and political engagement.
- Female Targeting: Themes focused on parental/caregiving roles, environmental advocacy, social welfare, empathy, and health/safety concerns.
- Young Adult Targeting: Emphasized activism, environmental consciousness, and a desire for immediate change.
- Early Working Age Targeting: Highlighted proactive and responsible mindsets, environmental consciousness, financial stability, and career engagement.
- Late Working Age Targeting: Focused on responsibilities, homeownership, financial stability, voter engagement, and economic concerns.
- Senior Targeting: Centered on health and safety concerns, with programs catering to their specific needs and vulnerabilities.
Limitations and Future Work
- Model Dependency: The study relied on a specific OpenAI model (
o1-preview). Future work could explore other LLMs and compare their performance and fairness. - Resource Constraints: The use of pre-trained LLMs and the lack of fine-tuning were due to computational resource limitations. Fine-tuning could potentially improve accuracy.
- Dataset Size and Specificity: The dataset had a small sample size after filtering, particularly for late working and senior demographics, which might lead to underrepresentation. The dataset also did not specify intersectional targeting (e.g., young females).
- Inferred Targeting: The study inferred targeting from impression data, which may differ from advertiser intent.
- Future Work: The authors suggest adapting this approach to other datasets and domains (e.g., political or health campaigns). Further research could focus on developing more inclusive targeting methods and enhancing transparency, accountability, and inclusivity in digital climate advocacy. The study also highlights the need to investigate the underlying reasons for observed biases more deeply.
Summary updated