TL;DR
This paper presents a rigorous replication study that challenges the effectiveness of Cohort Comfort Models (CCMs) for predicting building occupant thermal preferences. The authors demonstrate that previously reported performance gains were likely artifacts of methodological flaws—including information leakage and the use of metrics insensitive to class imbalance—and that CCMs fail to outperform simple, trivial baselines when evaluated correctly.
What Problem Does This Paper Address?
Predicting individual thermal comfort is a "holy grail" for building energy efficiency and occupant satisfaction. Standard models like the Predicted Mean Vote (PMV) work for groups but fail individuals. While Personal Comfort Models (PCMs) are more accurate, they require a burdensome amount of data (60+ surveys) from every single person.
Cohort Comfort Models (CCMs) were proposed as a solution: group similar people together so that a new occupant (a "cold start") can benefit from the data of their "cohort." However, this paper investigates whether these models actually provide a predictive edge or if the field has been misled by optimistic results stemming from poor machine learning practices.
Methods
The authors performed a multi-level validation:
- Methodological Assessment: They audited the original CCM study, identifying issues like inadequate cross-validation and information leakage (e.g., accidentally using the dataset index as a predictive feature).
- Revised Experimental Setup: They replaced randomized 80/20 splits with leave-one-out cross-validation to better handle small datasets and added F1-macro as a primary metric to account for the fact that most people usually feel "fine" (the "no change" class), making "warmer" or "cooler" preferences rare.
- Baseline Comparison: They introduced trivial baselines, such as a "Majority Class" predictor and an "ApproxDistribution" predictor (which guesses based on the frequency of past responses), to see if the fancy ML models were actually learning anything useful.
Datasets
- DORN: The original dataset from the 2023 study (20 participants, 6 months of data from Singapore).
- PersonAI: A new, independently collected dataset introduced by the authors (72 participants, 4 weeks of data from Austria). This dataset utilized Apple Watches, Oura Rings, and specialized air quality sensors to capture physiological and environmental context.
Key Insights and Ideas
- The Metric Mirage: The original study relied on F1-micro (essentially accuracy). In datasets where 70-90% of responses are "no change," a model can get a high score just by always predicting "no change." F1-macro, which weights all classes equally, reveals that these models are almost useless at predicting the minority classes ("warmer" or "cooler") that actually matter for HVAC control.
- Baselines are Essential: The authors found that a random predictor adhering to the target distribution often outperformed the cohort models. If a "dumb" random guesser beats your AI, your AI hasn't learned the underlying physics or psychology of comfort.
- Information Leakage: The study identified a bug in the original codebase where the dataframe index was included as a feature. In time-series data, the index often correlates with time of day or season, allowing the model to "cheat" by memorizing temporal patterns rather than learning comfort preferences.
- The Outlier Effect: The perceived "gains" of CCMs were often driven by a tiny number of participants with extreme class imbalances, rather than a general improvement for the typical user.
Key Results
- Failure to Beat Baselines: Under the F1-macro metric, all cohort models achieved lower median scores than a simple random baseline.
- Performance Collapse: For the DORN dataset, switching from F1-micro to F1-macro saw the General Purpose model's score drop from ~0.52 to ~0.28.
- Individual Models Still Lead: Personal Comfort Models (PCMs) remained the best-performing approach (median F1-macro ~0.41–0.51), but even they struggled to show clear convergence, suggesting that even 100+ data points might not be enough for stable prediction in the field.
Limitations and Future Work
The primary limitation is the inherent nature of field data: occupants rarely experience extreme thermal discomfort, leading to massive class imbalance. The authors suggest that future research should:
- Use Controlled Conditions: To obtain enough observations for "warmer" and "cooler" preferences, researchers may need to intentionally vary indoor temperatures.
- Adopt Rigorous ML Standards: Future studies must include trivial baselines, use imbalance-aware metrics, and strictly avoid information leakage.
- Re-evaluate Data Requirements: The widely cited "60 data points" rule for personal models appears insufficient; more work is needed to determine what actually constitutes a "stable" individual profile. Not discussed in the paper is how these findings might change with more advanced architectures like Transformers, though the data scarcity issue remains the fundamental bottleneck.
Summary updated