TL;DR
This paper compares two automated methods for analyzing online museum reviews: a "top-down" approach guided by policy-maker definitions and a "bottom-up" approach that discovers dimensions from the text itself. The "bottom-up" method revealed richer insights into visitor perceptions, highlighting the limitations of pre-defined quality frameworks.
What Problem Does This Paper Address?
The cultural tourism sector is increasingly interested in using online reviews to understand visitor experiences and assess service quality. While many studies use manual content analysis of these reviews, automated approaches are gaining traction. However, there's a lack of research comparing different automated methods for identifying quality dimensions in museums. This paper addresses this gap by comparing a "top-down" approach (using pre-defined categories from policymakers) with a "bottom-up" approach (discovering categories from the review text itself) to see which is more effective for evaluating museum quality from user-generated content.
Methods
The study employs two distinct automated text analysis approaches:
- "Top-Down" Approach: This method uses a supervised, keyword-based classifier. It relies on a predefined set of five quality dimensions (Ticketing and Welcoming, Space, Comfort, Activities, Communication) established by the Italian Ministry of Cultural Heritage and Activities and Tourism. Keywords associated with each dimension are used to classify reviews. This approach is essentially an automated version of manual coding based on expert-defined criteria.
- "Bottom-Up" Approach: This method uses an unsupervised topic modeling technique, specifically Latent Dirichlet Allocation (LDA). LDA identifies latent "topics" or themes within the text without any prior assumptions about what those themes should be. The identified topics are then interpreted to represent quality dimensions as perceived by the reviewers themselves.
Datasets
The study utilizes a dataset of 14,250 online reviews collected from the TripAdvisor pages of 100 Italian state museums during 2019. The reviews were filtered to include only those written in Italian to focus on local visitor perceptions.
Key Insights and Ideas
- Policy vs. Perception Mismatch: The "top-down" approach, based on policy maker guidelines, missed a significant portion of what visitors actually discuss. A large percentage of reviews (63%) were categorized as "Other Aspects," indicating that the predefined dimensions didn't capture the full spectrum of visitor feedback.
- "Bottom-Up" Uncovers Hidden Dimensions: The "bottom-up" LDA approach revealed three broader quality dimensions: Museum Cultural Heritage, Personal Experience, and Museum Services. Crucially, it showed that visitors prioritize cultural heritage (46%) and personal experiences (31%) more than just museum services (23%).
- Beyond Service Quality: The findings strongly suggest that for museums, quality evaluation needs to extend beyond just service-related aspects (like ticketing or staff) to encompass the historical and cultural significance of the exhibits and the emotional impact of the visit.
- Data Analytics Approach Matters: The choice of analytical method significantly impacts the insights gained. Policymakers and managers need to be aware of these differences to avoid information bias and gain a more holistic understanding of visitor feedback.
Key Results
- The "top-down" keyword-based classifier achieved an average accuracy of 80% and a recall of 50% across its five predefined categories.
- The "bottom-up" LDA approach identified 13 latent topics, which were then grouped into three main quality dimensions:
- Museum Cultural Heritage: 46% average probability.
- Personal Experience: 31% average probability.
- Museum Services: 23% average probability.
- When analyzing reviews classified as "Other Aspects" by the "top-down" method, the "bottom-up" approach still found a high probability of discussing Museum Cultural Heritage (48%) and Personal Experience (31%), with Museum Services being less prominent (21%).
Limitations and Future Work
- Language and Geographic Bias: The study is limited to Italian reviews, which may not represent the perceptions of international visitors. Future work could analyze reviews in multiple languages to capture broader perspectives.
- Museum Type: The study focused on Italian state museums. Expanding to include museums with different governance forms (foundations, corporate museums) or in other countries could enhance generalizability.
- Dynamic Nature of Quality: The authors acknowledge that quality dimensions can be emergent and vary by user and time period. Further investigations across different user groups or timeframes are suggested.
Summary updated