TL;DR
This paper investigates how well large language models (LLMs) like BERT and GPT understand brand perceptions and "affect associations" (e.g., "fun," "innovative"). They found that LLMs do capture some brand-affect relationships, but these don't always align with consumer perceptions, suggesting potential biases and limitations in how LLMs represent brands.
What Problem Does This Paper Address?
Understanding how consumers perceive brands is crucial for marketing. Traditionally, this involves time-consuming surveys. With the rise of digital content and powerful LLMs trained on vast amounts of text, researchers are exploring whether these models implicitly store brand perception knowledge. The challenge is to quantify and analyze this stored knowledge, particularly the "affect associations" – the emotional or qualitative attributes linked to brands (e.g., "Mountain Dew" being "adventurous"). This paper aims to bridge the gap between LLM capabilities and brand perception analysis, investigating if LLMs encode biases that could impact downstream applications like recommender systems or sentiment analysis.
Methods
The researchers employed a multi-pronged approach to probe LLMs for brand-affect associations:
- Contextual Geometry: They analyzed word representations across different layers of the LLMs to understand how brands and affect attributes are positioned in the embedding space. This involved measuring "affect-similarity" (how closely a brand and attribute word are represented) and "intra-brand/attribute similarity" (how similar brands or attributes are to each other).
- Implicit Association Tests (IAT): They adapted established psychological tests like the Word Embedding Association Test (WEAT) and introduced a log-probability bias score and Relational Inner Product Association (RIPA) to measure implicit biases and associations between brand categories and affect attributes.
- Probing Tasks: They used Masked Language Model (MLM) tasks, where the LLM predicts a masked word in a sentence. If the LLM correctly predicts an affect attribute when given a brand and context, it suggests the model has captured that association. They also explored a fine-tuned version (MLP-MLM) to assess the impact of training.
- Brand Archetypes: They mapped LLM-derived brand attributes to established archetypes (e.g., "Creator," "Jester") to see if LLMs capture these deeper brand personality dimensions.
Datasets
The study utilized several datasets:
- Ads. Dataset: A dataset of 35k "Action Reason" pairs, likely containing examples of brand-attribute relationships in advertisements.
- BCD Dataset: Consisting of 1962 sentences from webpages that contain both brand and affect attributes.
- Synthetic Dataset: 40 hand-crafted sentences designed to elicit specific brand-attribute associations.
- Lovett et al. (2014) Data: A dataset of 697 major US national brands and their associated perceptions and attributes, used to define affect attributes and brand categories. This data was also used to compare LLM findings against consumer perceptions.
Key Insights and Ideas
- LLMs Encode Affect Associations: The core finding is that LLMs do indeed capture associations between brands and affect attributes, confirming RQ1. This suggests LLMs are not just processing text but are learning underlying semantic relationships.
- Consistency Across LMs (RQ3): Many of these brand-affect associations are consistently observed across different LLMs, even those with varying architectures and training data. This points to a shared understanding of these relationships within the pre-training corpora.
- Bias in Associations (RQ2): The study found that these associations can exhibit biases, which might be amplified by fine-tuning. This is a critical implication for applications where these biases could lead to unfair or skewed outcomes.
- Mismatch with Consumer Perception (RQ4 & RQ5): A significant finding is that the brand personality or image captured by LLMs often does not align with either the intended brand personality or how consumers perceive the brand. This highlights a gap between what LLMs "know" and what is desired or perceived in the real world.
- Context Matters (but not always): While context influences affect associations, the study also found that some attributes are moderately influenced, suggesting that the LLM's internal representations are already quite rich.
Key Results
- Affect Similarity: The study found interesting associations, such as "Disney" being strongly associated with "magical" and "fun" across multiple LMs. Conversely, unexpected negative associations were also found (e.g., "Intel" and "Samsung" not being associated with "efficient").
- Contextualization: Word representations become more context-specific in the upper layers of LLMs, with RoBERTa showing the lowest self-similarity (indicating higher contextualization).
- Implicit Association Tests: The WEAT and Log Probability tests revealed significant associations between brand categories and Aaker's dimensions, with some consistent across LMs. For instance, the "Cars" brand category showed consistent rankings across all tested LMs.
- Brand Archetypes: LLMs struggled to distinctly capture brand archetypes. While they could identify top archetypes, the overlap with manually mapped attributes was consistently low, suggesting a lack of deep understanding of these nuanced brand personalities.
- Fine-tuning Impact: Fine-tuning improved affect associations, indicating that LLMs can learn and enhance these relationships with task-specific data. BERT and RoBERTa showed the most significant improvements.
Limitations and Future Work
- Data Limitations: The authors acknowledge that the current understanding of brand archetypes is limited by the available data and suggest that archetype-specific datasets are needed for more robust analysis. They also note that more data is required to fully understand the impact of fine-tuning and context.
- Negative Attributes: The analysis primarily focused on positive affect attributes, with only two negative attributes considered. Future work should explore the complete brand perception by including negative attributes.
- Consumer Perception Alignment: The significant mismatch between LLM-derived brand perceptions and consumer perceptions (RQ4 & RQ5) is a key area for future research. Understanding why this mismatch exists and how to bridge it is crucial.
- Broader Entities: The authors suggest extending this investigation to other iconic entities like sports teams, celebrities, and politicians.
- Task-Specific Data: The paper emphasizes the need for task-specific data to further enhance affect associations and refine brand-persona and archetype definitions.
Summary updated