TL;DR
This paper corrects an error in a previous study's use of "noise ceilings" for human brain data in representational similarity analysis (RSA). By developing a more appropriate cross-validation method, they show that the best-performing computational models do not fully explain human brain representations, a claim that was overstated in the original work.
What Problem Does This Paper Address?
Representational Similarity Analysis (RSA) is a powerful technique for comparing computational models of brain function to actual brain data. A key concept in RSA is the "noise ceiling," which represents the upper bound of performance achievable by any model, given the inherent noise and variability in the brain data. In a 2014 paper by Khaligh-Razavi and Kriegeskorte, noise ceilings were used to evaluate how well computational models explained human inferior temporal (IT) cortex representations. However, this addendum reveals an error in how these noise ceilings were calculated for the human data. Specifically, averaging human data across participants before calculating the noise ceiling masked the true variability and provided an inaccurate upper bound. This made it seem like models were performing closer to the ceiling than they actually were, potentially leading to overstating the explanatory power of the models. This paper aims to correct this error and provide a more accurate assessment of model performance against human brain data.
Methods
The authors introduce a novel and flexible cross-validation method to estimate both model performance and appropriate noise ceilings for fitted and unfitted models. This method involves:
- Stimulus-based bootstrapping: Generating 1000 bootstrap samples of stimuli to account for stimulus variability.
- Stimulus-crossvalidation: Within each bootstrap sample, 20 folds are created by randomly assigning 8 unique stimuli to a test set, ensuring no overlap with the training set.
- Subject-crossvalidation: For each stimulus fold, 4 folds are created by leaving out one subject at a time.
- Model Fitting and Evaluation: For each subject-crossvalidation fold, training data (from the remaining subjects and stimuli) is used to fit parameterized models. Predicted RDMs are then generated for the test stimuli.
- Performance Calculation: Model performance is measured by the correlation between the model-predicted RDM and the actual RDM from the test subject's data.
- Noise Ceiling Calculation: The noise ceiling (both upper and lower bounds) is calculated by correlating the test RDM with the average test-image RDM across all subjects (upper bound) or the training subjects (lower bound).
This procedure is repeated across all folds and bootstrap samples, with the final estimates being the average of these measures. This approach allows for unbiased estimation of model performance and noise ceilings, even for models with parameters that need to be fitted.
Datasets
The study uses human inferior temporal (hIT) cortex data, as described in Khaligh-Razavi and Kriegeskorte (2014). The paper does not introduce new datasets but re-analyzes existing human fMRI data.
Key Insights and Ideas
- The original noise ceiling calculation was flawed: Averaging human data before calculating the noise ceiling provided an artificially high upper bound, misrepresenting model performance.
- A more robust cross-validation is needed for fitted models: Parameterized models require careful cross-validation across both stimuli and subjects to obtain unbiased performance estimates and noise ceilings.
- The best computational models don't fully explain human IT representations: The corrected analysis shows that even the top-performing models do not reach the true noise ceiling, suggesting that our current computational models still have room for improvement in capturing the complexity of human brain representations.
- The claim of "fully explaining" human IT data was overstated: The original paper's strong claim is now tempered by the more accurate performance estimates.
Key Results
- The best-performing model (a weighted combination of AlexNet layers and category readouts) does not reach the lower bound of the corrected noise ceiling.
- However, this best-performing model is not statistically significantly below the lower bound of the noise ceiling, meaning it performs comparably to the best possible model given the data's inherent variability.
- Figures 1 and 2 in the paper present the corrected performance estimates for various deep supervised and not-strongly-supervised models, respectively, showing their correlations with human IT data and their respective noise ceilings.
Limitations and Future Work
- Limited number of subjects: The study notes that bootstrapping was limited to stimuli because there were only four subjects in the dataset. A larger number of subjects would allow for more robust subject-based bootstrapping.
- Distributional assumptions avoided: The authors explicitly state they avoided making distributional assumptions, which is a strength of their method. However, future work could explore methods that incorporate such assumptions if they lead to further refinements.
- Further exploration of model limitations: While the paper corrects an error and re-evaluates model performance, future work could delve deeper into why current models fall short of the noise ceiling and what specific aspects of human IT representations remain unexplained.
Summary updated