Replication of Thermal Comfort Cohort Models: No Support for Improved Thermal Preference Prediction

Thomas Hirsch, Katja Ceplak, Veronika Windisch, Samina Kadkhoda Masoumali, Matias Quintana, Clayton Miller, Stefano Schiavon, Gerald SchweigerarXiv 2026 · arXiv

TL;DR

This paper presents a rigorous replication study that challenges the effectiveness of Cohort Comfort Models (CCMs) for predicting building occupant thermal preferences. The authors demonstrate that previously reported performance gains were likely artifacts of methodological flaws—including information leakage and the use of metrics insensitive to class imbalance—and that CCMs fail to outperform simple, trivial baselines when evaluated correctly.

What Problem Does This Paper Address?

Predicting individual thermal comfort is a "holy grail" for building energy efficiency and occupant satisfaction. Standard models like the Predicted Mean Vote (PMV) work for groups but fail individuals. While Personal Comfort Models (PCMs) are more accurate, they require a burdensome amount of data (60+ surveys) from every single person.

Cohort Comfort Models (CCMs) were proposed as a solution: group similar people together so that a new occupant (a "cold start") can benefit from the data of their "cohort." However, this paper investigates whether these models actually provide a predictive edge or if the field has been misled by optimistic results stemming from poor machine learning practices.

Methods

The authors performed a multi-level validation:

  1. Methodological Assessment: They audited the original CCM study, identifying issues like inadequate cross-validation and information leakage (e.g., accidentally using the dataset index as a predictive feature).
  2. Revised Experimental Setup: They replaced randomized 80/20 splits with leave-one-out cross-validation to better handle small datasets and added F1-macro as a primary metric to account for the fact that most people usually feel "fine" (the "no change" class), making "warmer" or "cooler" preferences rare.
  3. Baseline Comparison: They introduced trivial baselines, such as a "Majority Class" predictor and an "ApproxDistribution" predictor (which guesses based on the frequency of past responses), to see if the fancy ML models were actually learning anything useful.
Figure 1: Overview of the study methodology.

Datasets

  • DORN: The original dataset from the 2023 study (20 participants, 6 months of data from Singapore).
  • PersonAI: A new, independently collected dataset introduced by the authors (72 participants, 4 weeks of data from Austria). This dataset utilized Apple Watches, Oura Rings, and specialized air quality sensors to capture physiological and environmental context.

Key Insights and Ideas

  • The Metric Mirage: The original study relied on F1-micro (essentially accuracy). In datasets where 70-90% of responses are "no change," a model can get a high score just by always predicting "no change." F1-macro, which weights all classes equally, reveals that these models are almost useless at predicting the minority classes ("warmer" or "cooler") that actually matter for HVAC control.
  • Baselines are Essential: The authors found that a random predictor adhering to the target distribution often outperformed the cohort models. If a "dumb" random guesser beats your AI, your AI hasn't learned the underlying physics or psychology of comfort.
  • Information Leakage: The study identified a bug in the original codebase where the dataframe index was included as a feature. In time-series data, the index often correlates with time of day or season, allowing the model to "cheat" by memorizing temporal patterns rather than learning comfort preferences.
  • The Outlier Effect: The perceived "gains" of CCMs were often driven by a tiny number of participants with extreme class imbalances, rather than a general improvement for the typical user.

Key Results

  • Failure to Beat Baselines: Under the F1-macro metric, all cohort models achieved lower median scores than a simple random baseline.
  • Performance Collapse: For the DORN dataset, switching from F1-micro to F1-macro saw the General Purpose model's score drop from ~0.52 to ~0.28.
  • Individual Models Still Lead: Personal Comfort Models (PCMs) remained the best-performing approach (median F1-macro ~0.41–0.51), but even they struggled to show clear convergence, suggesting that even 100+ data points might not be enough for stable prediction in the field.

Limitations and Future Work

The primary limitation is the inherent nature of field data: occupants rarely experience extreme thermal discomfort, leading to massive class imbalance. The authors suggest that future research should:

  • Use Controlled Conditions: To obtain enough observations for "warmer" and "cooler" preferences, researchers may need to intentionally vary indoor temperatures.
  • Adopt Rigorous ML Standards: Future studies must include trivial baselines, use imbalance-aware metrics, and strictly avoid information leakage.
  • Re-evaluate Data Requirements: The widely cited "60 data points" rule for personal models appears insufficient; more work is needed to determine what actually constitutes a "stable" individual profile. Not discussed in the paper is how these findings might change with more advanced architectures like Transformers, though the data scarcity issue remains the fundamental bottleneck.

Summary updated

Abstract

Cohort comfort models (CCMs) aim to overcome the extensive individual data requirements of personal comfort models (PCMs) by grouping occupants based on demographic and psychometric characteristics (cold start) or similar thermal preferences (warm start). We evaluated the validity of the reported CCM performance gains through (i) a methodological assessment of the original study, (ii) a replication using one of the two original datasets (DORN: 20 participants, 6 months) under a revised experimental protocol, and (iii) a replication using an independently collected dataset (PersonAI: 72 participants, 4 weeks). Classification performance was evaluated using F1-micro, as in the original study, and F1-macro to account for class imbalance. Under F1-micro, a trivial baseline that always outputs the majority class performed on par with cohort models. Under F1-macro, all cohort models achieved lower median scores than a simple baseline that makes random predictions based on the estimated distribution of thermal preferences. In our methodological assessment we identified the following key issues: Limited data per participant and per minority class, reliance on a metric insensitive to class imbalance, missing trivial baselines, inadequate cross validation strategy, and multiple occurrences of information leakage. Our findings challenge the original study's conclusion that CCMs improve thermal preference prediction. CCMs did not outperform simple class-distribution baselines. Future studies should improve experimental designs, including controlled thermal conditions, to obtain sufficient observations across thermal preference classes.
BibTeX
@misc{hirsch2026replication,
  author = {Thomas Hirsch and Katja Ceplak and Veronika Windisch and Samina Kadkhoda Masoumali and Matias Quintana and Clayton Miller and Stefano Schiavon and Gerald Schweiger},
  title = {{Replication of Thermal Comfort Cohort Models: No Support for Improved Thermal Preference Prediction}},
  year = {2026},
  eprint = {2611.12178},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12178},
}
Figure