TL;DR
This paper introduces a novel method using ChatGPT and Retrieval Augmented Generation (RAG) to automatically extract and aggregate public impressions of celebrities from social media (X/Twitter) and then find the underlying reasons for these impressions from web pages. This approach aims to provide fans with a more comprehensive understanding of celebrity perceptions, overcoming limitations of previous methods.
What Problem Does This Paper Address?
Understanding public sentiment and opinions about celebrities, especially from the vast and often implicit information shared on social media like X (formerly Twitter), is challenging. While microblogs capture immediate reactions, the reasons behind these impressions are frequently buried in external context like news articles or advertisements, making them hard to synthesize. Existing methods often struggle with the nuance and implicit nature of these opinions, or are limited in their ability to comprehensively collect and aggregate them. This paper tackles the problem of bridging the gap between surface-level social media impressions and the deeper, contextualized reasons behind them, aiming to empower fans with a richer understanding.
Methods
The paper proposes a two-stage approach:
- Collecting and Aggregating Celebrity Impressions:
- Posts from X (formerly Twitter) related to specific celebrities are collected using their names as search queries.
- ChatGPT (specifically
gpt-4-turbo-2024-04-09) is then used to analyze these posts, identifying "aspects" (e.g., acting skills, appearance) and the "impressions" (e.g., amazing, cute) associated with them. - The model aggregates these into "aspect-impression pairs." This stage leverages ChatGPT's natural language understanding to go beyond simple keyword matching and capture more nuanced relationships.
- Collecting and Aggregating Reasons for Impressions:
- The identified aspect-impression pairs are used as keywords to search for relevant web pages.
- The top 30 web pages for each keyword are collected.
- ChatGPT, enhanced by the Retrieval Augmented Generation (RAG) framework, is employed to extract detailed reasons for the impressions from these web pages. RAG is crucial here as it grounds ChatGPT's responses in the retrieved web content, mitigating hallucinations and improving reliability.
- Finally, these collected reasons are aggregated and ranked by frequency, providing a structured overview of why certain impressions exist.
Datasets
The paper uses publicly available X (formerly Twitter) posts and web pages as its data sources. For evaluation, a manually annotated reference dataset was created for both the impression collection/aggregation and reason collection/aggregation tasks. The authors selected 10 celebrities who are frequently discussed on X for their data collection.
Key Insights and Ideas
- ChatGPT's Versatility: The study highlights ChatGPT's effectiveness not only in summarizing and extracting information but also in its ability to aggregate and categorize complex sentiments and reasons, surpassing previous methods.
- RAG for Reliability: The integration of RAG significantly enhances the trustworthiness of the extracted reasons by forcing the LLM to rely on retrieved web content rather than just its internal knowledge, thus reducing the risk of hallucination.
- Bridging Microblogs and Web Content: The core innovation lies in effectively linking the immediate, often implicit, impressions from social media to the more detailed, contextualized explanations found on the broader web.
- Beyond Simple Patterns: The method moves beyond rigid "celebrity's X" patterns, allowing for a more flexible identification of celebrity aspects and associated sentiments.
Key Results
- Impression Aggregation: Manual evaluation showed ChatGPT achieved high accuracy in collecting and aggregating celebrity impressions, with a micro-average recall of 0.58 and precision of 0.74 for collection, and recall of 0.61 and precision of 0.71 for aggregation.
- Reason Collection: ChatGPT demonstrated high performance in collecting reasons from web pages, achieving recall, precision, and F1-scores around 0.9. It was notably robust to noisy web page retrieval errors.
- Comparison with mT5: In automatic evaluations for reason detection and collection, ChatGPT outperformed mT5 across all metrics, particularly in precision for detection. For reason collection, ChatGPT generated results closer to the reference compared to mT5.
- Manual Reason Aggregation: With examples provided, ChatGPT achieved a recall of 0.77, precision of 0.88, and F1-score of 0.81 in reason aggregation, outperforming the "without examples" scenario in recall and F1, though with slightly higher redundancy.
Limitations and Future Work
- LLM Hallucinations: Despite RAG, LLMs can still generate plausible but incorrect information. The authors acknowledge this inherent risk.
- Keyword Selectivity: The approach relies on aspect-impression pairs that can be directly used as search keywords. For pairs that yield insufficient web pages, more advanced techniques like semantic or embedding search are needed.
- Reproducibility: Manual data collection and processing steps may pose challenges for exact reproducibility.
- Future Work: The authors suggest comparing ChatGPT with larger, more advanced LLMs (e.g., Mistral Large, LLaMA 3 70B) for a more comprehensive evaluation. They also propose exploring more sophisticated information retrieval techniques for keyword pairs that don't yield enough web pages. The ethical implications of such tools and the potential for reinforcing biases are also highlighted as areas for ongoing consideration.
Summary updated