Price Prediction in a Trading Agent Competition

Michael P. Wellman, Daniel M. Reeves, Kevin M. Lochner, Yevgeniy Vorobeychik

TL;DR

This paper analyzes how agents in the 2002 Trading Agent Competition (TAC) predicted hotel prices, a crucial factor for success in the travel shopping domain. They found that incorporating game-specific information, particularly flight prices, significantly improved prediction accuracy, even outperforming machine learning methods with a purely analytical approach based on competitive equilibrium.

What Problem Does This Paper Address?

The Trading Agent Competition (TAC) is a simulated market where agents act as travel agencies, assembling trips for clients by bidding on flights, hotels, and entertainment. A key challenge in this domain is predicting future hotel prices, which are not finalized until auctions close. These prices heavily influence an agent's ability to choose optimal itineraries and bid effectively, directly impacting their overall score. While many agents use price prediction, the paper aims to survey the diverse approaches taken and analyze their effectiveness, particularly in relation to the information they utilize.

Methods

The authors surveyed participants of the TAC-02 competition to understand their price prediction strategies. They then analyzed the agents' actual predictions from the finals and semifinals using two novel evaluation metrics: Euclidean distance (measuring absolute accuracy) and Expected Value of Perfect Prediction (EVPP), which quantifies how well a prediction supports optimal decision-making. They also introduced a new measure for relating prediction accuracy to overall game performance. To further investigate the impact of specific information sources, they created variants of one agent (Walverine) that either ignored client data or flight prices.

Datasets

The primary dataset consists of the price predictions made by agents in the TAC-02 finals and semifinals. This data was collected through a survey distributed to the competition participants. The authors also used data from the TAC-01 tournament for comparative analysis and created controlled trials with modified versions of the Walverine agent to isolate the impact of specific features.

Key Insights and Ideas

  • Flight prices are gold: Incorporating game-specific information, especially initial flight prices, proved to be a major differentiator for accurate hotel price prediction.
  • Analytical beats empirical (sometimes): A purely analytical approach based on competitive equilibrium analysis achieved accuracy comparable to the best machine learning methods, even without historical data. This suggests that idealized economic models can be surprisingly effective in complex market simulations.
  • EVPP: A better measure of value: The newly introduced Expected Value of Perfect Prediction (EVPP) metric better captures the practical utility of price predictions for decision-making, going beyond simple absolute accuracy.
  • Instance-specific info is key: Agents that leveraged information unique to each game instance (like flight prices) generally outperformed those relying solely on historical averages.
  • Machine learning is good, but not always best: While machine learning methods were effective at learning relationships between flight and hotel prices, a theoretically grounded analytical approach could match or exceed their performance.

Key Results

  • The three best price predictors (ATTac01, Walverine, and kavayaH) were precisely those that incorporated flight prices into their predictions.
  • Walverine's analytical approach, based on competitive equilibrium, achieved accuracy comparable to the best machine learning agent (ATTac01).
  • Ignoring flight prices significantly degraded Walverine's prediction quality (increasing EVPP from 38.0 to 47.9).
  • Both Euclidean distance and EVPP were highly correlated with overall game scores, indicating that prediction accuracy translates to performance.
  • The EVPP coefficient in regressions with game scores ranged from -8.89 to -11.59, suggesting that reducing EVPP by one point could lead to an increase of roughly eight score points.

Limitations and Future Work

The authors acknowledge several limitations:

  • Focus on initial prediction: The study primarily focused on initial price prediction, while many agents placed more emphasis on interim prediction (updating predictions as the game progresses).
  • Abstraction of predictions: The analysis often reduced complex prediction outputs (like probability distributions or pricelines) to simpler forms (means or baseline prices), which might not fully capture the agents' intended use.
  • Interconnectedness of strategies: It's challenging to isolate prediction strategy from an agent's overall decision-making and trading strategy, making a complete evaluation difficult.

Future work could explore interim prediction strategies more deeply, develop measures tailored to specific agent prediction methods, and investigate how prediction strategies interact with other components of an agent's overall behavior. The authors also suggest that research competitions like TAC are valuable for generating knowledge and insights that might be harder to achieve through independent research.

Summary updated

Abstract

The 2002 Trading Agent Competition (TAC) presented a challenging market game in the domain of travel shopping. One of the pivotal issues in this domain is uncertainty about hotel prices, which have a significant influence on the relative cost of alternative trip schedules. Thus, virtually all participants employ some method for predicting hotel prices. We survey approaches employed in the tournament, finding that agents apply an interesting diversity of techniques, taking into account differing sources of evidence bearing on prices. Based on data provided by entrants on their agents' actual predictions in the TAC-02 finals and semifinals, we analyze the relative efficacy of these approaches. The results show that taking into account game-specific information about flight prices is a major distinguishing factor. Machine learning methods effectively induce the relationship between flight and hotel prices from game data, and a purely analytical approach based on competitive equilibrium analysis achieves equal accuracy with no historical data. Employing a new measure of prediction quality, we relate absolute accuracy to bottom-line performance in the game.
BibTeX
@article{wellman2004price,
  author = {Michael P. Wellman and Daniel M. Reeves and Kevin M. Lochner and Yevgeniy Vorobeychik},
  title = {{Price Prediction in a Trading Agent Competition}},
  journal = {J. Artif. Intell. Res.},
  year = {2004},
  doi = {10.1613/jair.1333},
}

Loading related content...

Figure