TL;DR
This paper investigates the "anchoring and adjustment" cognitive bias in software estimation. The study found that providing estimators with an initial, potentially incorrect, estimate significantly influences their final estimate, leading to biased results. This bias impacts both experienced and novice estimators, suggesting a need for greater awareness and mitigation strategies in software project planning.
What Problem Does This Paper Address?
Software estimation is notoriously difficult, often leading to significant overruns or underruns. While various models and techniques exist, the human element remains a critical factor. This paper tackles the problem of cognitive biases, specifically "anchoring and adjustment," which can subtly but powerfully skew human judgment, particularly under uncertainty. The authors highlight that software estimation, being a judgment-based activity performed under uncertainty, is susceptible to such biases. They aim to empirically demonstrate how this specific bias affects software estimation and its implications.
Methods
The researchers conducted an experiment involving participants estimating the time required for a fictional software project. The experiment employed a between-subjects design with three conditions:
- Control Condition: Participants received no initial estimate.
- Low Anchor Condition: Participants were given a low initial estimate ("2 months").
- High Anchor Condition: Participants were given a high initial estimate ("20 months").
Participants were provided with a detailed project description and were free to use any estimation technique they preferred. After providing their estimate, they were also asked to provide a confidence interval for their estimate and a justification for their chosen method. The justifications were later analyzed to classify participants' estimation techniques as either model-based or expert-based.
Datasets
Not discussed in the paper. The study used a fictional software project description for the experiment.
Key Insights and Ideas
- Anchoring is Real in Software Estimation: The study provides strong empirical evidence that the anchoring and adjustment bias significantly impacts software estimates. Providing an initial number, even if arbitrary, pulls estimates towards it.
- High Anchors Have a Stronger Effect: The bias is more pronounced when a high anchor is provided, leading to substantially longer estimates compared to low anchors or no anchors.
- Adjustment is Insufficient: People tend to adjust away from the anchor but not enough to fully compensate for its influence, resulting in biased final estimates.
- Experienced Estimators Aren't Immune: Even individuals with prior experience in software estimation are susceptible to this bias, though the statistical significance was slightly weaker in this subgroup.
- Expert-Based Estimation is More Vulnerable: Estimators relying primarily on expert judgment (rather than models) showed stronger effects of the anchoring bias.
- Confidence Ranges Don't Fully Compensate: Providing a confidence interval for their estimates did not fully mitigate the anchoring bias; the range between the low and high anchor conditions remained significant even when considering worst-case and best-case scenarios.
- Estimators are Overconfident: The results suggest that estimators tend to be too confident in their own estimations, especially considering the magnitude of the observed bias.
Key Results
- Overall Impact: The mean estimate for the "20 months" anchor condition was 17.4 months, significantly higher than the control condition (8.3 months) and the "2 months" anchor condition (6.8 months).
- Low vs. High Anchor: The difference between the low anchor and high anchor conditions was statistically significant (p < 0.001).
- Low Anchor vs. Control: No significant difference was found between the low anchor and control conditions (p > 0.1). This suggests that low anchors might not have as strong an effect, or that participants in the control group might have been influenced by their own inherent optimism.
- High Anchor vs. Control: The high anchor condition resulted in significantly higher estimates than the control condition (p < 0.01).
- Experienced Estimators: For experienced estimators, the pattern held: "2 months" (7.8 months), control (9.0 months), and "20 months" (17.8 months). The difference between low and high anchors was still significant (p < 0.02).
- Expert-Based Estimators: This group showed the strongest effects, with means of 5.1 months ("2 months"), 7.8 months (control), and 15.4 months ("20 months"). The difference between low and high anchors was highly significant (p < 0.001).
- Model-Based Estimators: The small sample size for this group prevented statistically significant conclusions, though the trend suggested a similar pattern.
- Confidence Ranges: The mean best-case estimate for the "2 months" condition was 8.7 months, while the mean worst-case estimate for the "20 months" condition was 12.8 months. This difference was statistically significant (p < 0.05), indicating that confidence ranges did not fully compensate for the bias.
Limitations and Future Work
Limitations:
- Sample Heterogeneity: The participant pool was a mix of industry professionals and graduate students, with varying levels of experience.
- Task Representativeness: The experiment used a single, fictional project description, and the task of estimating in months might not perfectly mirror all real-world estimation scenarios.
- Social Bias: While efforts were made to minimize social influence, the possibility of participants reacting to social cues rather than purely cognitive bias exists.
- Anchor Strength: The chosen low anchor (2 months) might not have been sufficiently low to elicit a significant difference from the control group.
- Mono-Operation Bias: Using only one set of project specifications might limit the generalizability of the findings.
Future Work:
- Different Estimation Units: Explore the effect of anchors when estimates are requested in different units (e.g., weeks instead of months).
- Project Lifecycle Stages: Investigate how the anchoring effect changes at different stages of the project lifecycle as more detailed information becomes available.
- Mitigation Strategies: Further research into effective ways to shield estimators from anchors or to help them consciously counteract the bias.
- Broader Population: Replicate the experiment with a larger and more diverse sample of software estimators.
- Other Biases: Explore the interplay of anchoring with other cognitive biases in software estimation.
Summary updated