Introduction
In today's hyper-competitive market, capturing the "voice of the customer" is essential for strategic decision-making. Social media platforms offer a vast repository of consumer sentiment, yet extracting actionable insights remains a challenge. Traditional topic modeling techniques, such as Latent Dirichlet Allocation (LDA), often provide a static snapshot of discourse. These methods struggle to account for the fluid, time-varying nature of online conversations and frequently require significant manual intervention to interpret results.
This paper addresses these limitations by proposing a dynamic, automated framework that identifies the lifecycle of topics—their birth, growth, and eventual decay. By moving beyond retrospective analysis, the system identifies "emerging" topics that represent future business opportunities.
Methodology
The researchers introduce a two-stage approach that leverages semantic understanding rather than simple keyword frequency. The framework transitions from raw data collection to prioritized trend identification.
1. Dynamic Topic Detection
The first stage focuses on capturing the semantic meaning of social media posts:
- Vectorization: The system utilizes RoBERTa-based Sentence-BERT to transform text into high-dimensional vectors. This ensures that semantically related terms (e.g., "battery life" and "energy density") are grouped together even without shared keywords.
- Clustering: K-means clustering is applied to discrete time slices (e.g., annually). The optimal number of topics is determined automatically using Gap Statistics, allowing the model to adapt its resolution as data volume fluctuates.
- Topic Linking: Clusters are connected across time slices using a Jaccard-like intersect measure, enabling the visualization of topic evolution through Sankey-style diagrams.
2. Emergence Scoring and Prioritization
To identify promising trends, the latest time slice is evaluated against four key indicators:
- Novelty: The ratio of new keywords appearing in a topic.
- Growth: The increase in document volume over time.
- Community: The diversity of unique users participating in the discussion, filtering out "echo chambers."
- Coherence: The internal semantic consistency of the topic's keywords.
These indicators are aggregated using TOPSIS (Technique for Order of Preference by Similarity to Ideal Solution), a multi-criteria decision-making method that ranks topics based on their proximity to an "ideal" emerging trend.
Case Study: The Electric Vehicle Industry
The framework was tested using a massive dataset from the r/teslamotors subreddit, comprising over 2.8 million posts and comments from 2015 to 2019. The rapidly evolving Electric Vehicle (EV) sector served as an ideal environment to test the model's ability to catch shifting trends.
Key Insights
The model categorized topics into four distinct evolutionary paths:
- Constant: Steady mentions (e.g., "Cold climate" performance).
- Seed: Small topics that eventually branch into specialized areas.
- Consolidate: Merging of multiple old topics into a new focus (e.g., "Hardware acceleration" for autopilot).
- Temporary: Short-lived bursts of interest.
By using Pre-trained Language Models (PLMs), the system successfully tracked how customer interests diversified from general aesthetics to specific technical performance metrics.
Results and Implications
The system identified 138 distinct topics over the five-year study period. Quantitative validation showed that data peaks in the model aligned precisely with major real-world events, such as the announcements of the Tesla Model 3 and Model Y.
In the final 2019 time slice, the model identified "Exterior" as the top-ranking emerging topic (score: 0.8153), followed by "Regulation" and "Computer Vision Systems." These findings reflect a market shift toward design maturity and the increasing impact of government policy on EV adoption. Additionally, "Subsidies" was identified as a constant topic that was re-emerging, signaling renewed customer concern regarding affordability.
Limitations and Future Directions
While the framework is robust, it currently relies on noun chunks for pre-processing, which may overlook nuances in sentiment expressed through adjectives or verbs. Furthermore, the four emergence indicators were weighted equally in this study; future applications might benefit from industry-specific weightings. Moving from yearly to monthly or weekly time slices would also enhance the system's agility for real-time business intelligence.
Summary updated