TL;DR
This paper rigorously evaluates simple decision heuristics against state-of-the-art statistical learning methods for the "comparison" problem (choosing the best option from a set). Surprisingly, heuristics perform remarkably well, often only a few percentage points behind the best algorithms, highlighting their efficiency and effectiveness in real-world scenarios.
What Problem Does This Paper Address?
The paper tackles the fundamental problem of comparison: how do we choose the best option from a set of alternatives when the true value of each option is unknown? This is a ubiquitous challenge in human and animal decision-making, from venture capitalists choosing investments to bees selecting nest sites. Traditional approaches often rely on complex statistical models that require extensive data and computational resources. However, humans and animals frequently employ simpler "heuristics" – mental shortcuts that use minimal information and simple rules. The central question is: how effective are these simple heuristics when pitted against powerful, data-hungry statistical learning algorithms on real-world comparison tasks?
Methods
The authors compare three well-established decision heuristics – take-the-best, tallying, and single-cue – against a suite of sophisticated statistical learning algorithms. These include:
- Classification Algorithms: Random Forests, Support Vector Machines (SVMs) with linear and radial kernels, and Naive Bayes.
- Regression Algorithms: Multiple Linear Regression (MLR) with elastic net penalty and Random Forest Regression.
The core of their methodology involves training these models on various sizes of datasets and evaluating their performance using accuracy (correctly identifying the better option) and a linear loss metric. A key aspect of their approach is that they do not dichotomize attributes (convert them to binary values), a common practice in earlier heuristic research, believing this preserves more information and leads to fairer comparisons. They also learn cue directions and orders for the heuristics directly from the training data.
Datasets
The study utilizes a comprehensive benchmark of 56 diverse real-world datasets. These datasets span a wide range of domains, including biology, business, computer science, economics, and psychology. The datasets vary significantly in size, from 17 to 3,281 objects, and in the number of attributes, from 3 to 21. This extensive collection aims to provide a robust evaluation across various types of comparison problems. The datasets are publicly available.
Key Insights and Ideas
- Heuristics are surprisingly competitive: The most striking finding is that simple heuristics, despite their minimal information and computational requirements, perform nearly as well as complex statistical models. This challenges the notion that sophisticated algorithms are always superior.
- "Good enough" is often sufficient: The small performance gap between heuristics and top algorithms suggests that for many real-world comparison tasks, heuristics provide a "good enough" solution with vastly lower costs.
- Comparison is an "easy" problem: The authors speculate that the success of heuristics in comparison tasks might be due to the inherent nature of comparison itself being a relatively simpler problem compared to full-blown regression or classification.
- No need to dichotomize: The study's finding that models (including heuristics) generally perform better when attributes are not dichotomized suggests that earlier research might have underestimated heuristics by using this simplification.
Key Results
- Average Performance: On average, across the 56 datasets, heuristics were only a few percentage points behind the best-performing statistical algorithms. For instance, take-the-best was, on average, within 0.019 accuracy points of the top algorithm for larger training set sizes.
- Learning Curves: Heuristics often achieved the highest accuracy in all or parts of the learning curve, demonstrating their ability to learn effectively even with limited training data.
- Computational Efficiency: Heuristics were orders of magnitude faster to train and test than statistical models. Take-the-best and tallying required only 0.40 seconds, while SVMs could take hundreds of minutes.
- MLR's Strong Showing: Interestingly, Multiple Linear Regression (with elastic net) emerged as a top performer, especially with larger training sets, suggesting that linear models can be very powerful for comparison tasks.
Limitations and Future Work
- Focus on Comparison: The study is specifically focused on the "comparison" problem. The authors acknowledge that heuristics might not perform as well on other decision-making tasks like classification or regression.
- "Easy" Problem Hypothesis: The authors propose that comparison might be an inherently easier problem. Further theoretical work is needed to understand why this is the case.
- Future Directions: The paper calls for more theoretical investigation into the comparison problem and the development of statistical learning algorithms that specifically leverage its properties. They also encourage further study and mathematical analysis of decision heuristics.
Summary updated