Support Vector Machines with Manifold Learning and Probabilistic Space Projection for Tourist Expenditure Analysis

Xin Xu, Rob Law, Tom WuInternational Journal of Computational Intelligence Systems 2009

TL;DR

This paper tackles the challenge of analyzing complex, noisy tourism expenditure data by proposing a novel approach that combines Support Vector Machines (SVMs) with two feature projection techniques: ISOMAP for noise reduction and manifold learning, and a probabilistic space mapping for scale transformation. This integrated method significantly improves prediction accuracy and statistical coincidence for business traveler expenditure analysis.

What Problem Does This Paper Address?

The tourism industry generates substantial economic contributions, making it crucial to understand traveler behavior. However, analyzing raw tourism data, particularly expenditure patterns of business travelers, presents significant challenges. This data is often complex, nonlinear, and contains noise due to varied collection circumstances. Existing data mining techniques like rough sets and neural networks have shown limitations in achieving satisfactory accuracy and statistical coincidence for this type of data. While SVMs offer a powerful framework, their performance is highly dependent on feature selection and kernel function choices, which can be subjective and difficult to optimize. This paper aims to address these issues by developing a more robust and accurate method for classifying tourist expenditure data.

Methods

The core of this paper's contribution lies in its novel integration of two feature projection techniques with SVMs:

  • ISOMAP (Isometric Feature Mapping): This manifold learning algorithm is used for dimension reduction and, crucially, for data selection and noise reduction. By mapping high-dimensional data to a lower-dimensional space, ISOMAP helps visualize the data's intrinsic structure. The authors leverage this to identify and discard "superposed" or noisy training data points that are mixed across different classes, thereby simplifying the decision boundary for the SVM and improving its stability.
  • Probabilistic Space Projection (PSP): This technique is employed for scale transformation. Tourism data often contains features with different measurement scales (discrete vs. continuous). PSP transforms feature values into posterior probabilities of belonging to different classes. This creates a uniform representation, allowing distance measures to be more appropriate and improving the SVM's classification performance.

These two methods are applied sequentially: ISOMAP first for noise reduction and data selection, followed by PSP for scale transformation, before feeding the processed data into a multi-class SVM classifier. The paper utilizes a one-vs-all strategy for multi-class SVM implementation.

Datasets

The study utilizes a dataset of 1,282 non-transit visitors to Hong Kong collected in October 2005 by the School of Hotel & Tourism Management at The Hong Kong Polytechnic University. From this, 303 respondents identified as business travelers provided usable demographic and trip profile data. This data was used for classification, with expenditure amounts categorized into High (H), Medium (M), and Low (L) using an equal percentile approach. The dataset was split into five pairs of training (80%) and testing (20%) sets, denoted as HK-i (i=1,2,...,5), for a 5-fold cross-validation approach.

Key Insights and Ideas

  • Noise is a significant impediment: The paper strongly suggests that noise in tourism expenditure data is a primary reason for the poor performance of traditional methods and even standard SVMs. ISOMAP's ability to visualize and help remove this noise is a key insight.
  • Feature representation matters immensely: The study highlights that simply applying SVMs to raw data is insufficient. The way features are represented and scaled (via ISOMAP and PSP) has a profound impact on classification accuracy and efficiency.
  • Manifold learning for data cleaning: Applying manifold learning techniques like ISOMAP is not just for visualization or dimension reduction but can serve as a practical data cleaning tool by identifying and removing problematic data points.
  • Probabilistic transformation unifies scales: The probabilistic space projection offers an elegant solution to the problem of disparate feature scales, enabling more effective distance calculations within the SVM framework.

Key Results

The proposed method, combining ISOMAP and PSP with SVMs, demonstrated superior performance compared to baseline SVMs and other classification methods:

  • Improved Accuracy: The SVMs with ISOMAP-based data selection generally showed better classification results than SVMs without it. The full proposed method (SVMs with ISOMAP and PT features) achieved the highest test precision among the evaluated methods, reaching 58.3%.
  • Reduced Training Time: ISOMAP-based data selection not only improved accuracy but also significantly reduced training time, as noisy data makes SVM training more computationally intensive. For instance, training time for SVMs with ISOMAP was considerably lower than for standard SVMs.
  • Enhanced Statistical Coincidence: The paper emphasizes that the proposed method also improves statistical coincidence (agreement between estimated and actual values), as evidenced by Wilcoxon Signed Ranks tests, where SVMs with PT features showed a high probability (0.738) of coming from the same distribution as the actual data.

Limitations and Future Work

  • Parameter Sensitivity: While the paper demonstrates the effectiveness of the proposed method, the performance of ISOMAP is sensitive to its neighborhood parameter k, and the performance of SVMs is sensitive to kernel parameters like sigma (σ) and the regularization parameter C. Table 5 shows how correct rates vary with different σ values.
  • Subjectivity in ISOMAP: Although ISOMAP helps identify noisy data, the exact criteria for discarding "superposed" data might still involve some degree of subjective judgment or require further refinement.
  • Future Work: The authors suggest that further research could explore other manifold learning techniques and probabilistic projection methods to further enhance SVM performance. Investigating different multi-class SVM strategies beyond one-vs-all could also be beneficial.

Summary updated

Abstract

The significant economic contributions of the tourism industry in recent years impose an unprecedented force for data mining and machine learning methods to analyze tourism data. The intrinsic problems of raw data in tourism are largely related to the complexity, noise and nonlinearity in the data that may introduce many challenges for the existing data mining techniques such as rough sets and neural networks. In this paper, a novel method using SVMbased classification with two nonlinear feature projection techniques is proposed for tourism data analysis. The first feature projection method is based on ISOMAP (Isometric Feature Mapping), which is a class of manifold learning approaches for dimension reduction. By making use of ISOMAP, part of the noisy data can be identified and the classification accuracy of SVMs can be improved by appropriately discarding the noisy training data. The second feature projection method is a probabilistic space mapping technique for scale transformation. Experimental results on expenditure data of business travelers show that the proposed method can improve prediction performance both in terms of testing accuracy and statistical coincidence. In addition, both of the feature projection methods are helpful to reduce the training time of SVMs.
BibTeX
@article{xu2009support,
  author = {Xin Xu and Rob Law and Tom Wu},
  title = {{Support Vector Machines with Manifold Learning and Probabilistic Space Projection for Tourist Expenditure Analysis}},
  journal = {International Journal of Computational Intelligence Systems},
  year = {2009},
}
Figure