Abstract
Hypothesis tests and sure independence screening (SIS) procedure based on ball statistics, including ball divergence, ball covariance, and ball correlation, are developed to analyze complex data in metric spaces, such as shape, directional, compositional, and symmetric positive definite matrix data. The ball divergence and ball covariance based distribution-free tests are implemented to detect distribution differences and associations in metric spaces. Furthermore, several generic non-parametric feature selection procedures based on ball correlation (BCor-SIS and its variants) are implemented to tackle challenges in ultra-high dimensional data. A fast implementation for large-scale multiple K-sample testing with ball divergence is supported, which is particularly helpful for genome-wide association studies (GWAS).
Introduction
Modern data analysis frequently encounters "complex" data that defies traditional statistical assumptions. Linear models and normality assumptions often fail when dealing with non-linear dependencies or data residing in non-Euclidean metric spaces, such as directional data (angles), compositional data (percentages), or 3D shapes. Additionally, the "curse of dimensionality" in fields like genomics creates a "needle in a haystack" problem where thousands of irrelevant variables must be pruned to identify meaningful predictors.
The R package 'Ball' addresses these challenges by implementing a robust, non-parametric framework based on "ball statistics." These methods provide a unified approach for hypothesis testing and feature selection that is both distribution-free and model-free.
Methodology: The Ball Statistics Framework
The core of the package lies in three primary statistics:
- Ball Covariance (BCov) and Ball Correlation (BCor): These are generic measures of dependence. Unlike the Pearson correlation coefficient, BCor is zero if and only if the variables are truly independent. This property allows it to capture complex, non-linear relationships that traditional methods might miss.
- Ball Divergence (BD): This statistic measures the distance between probability distributions. It is zero if and only if two distributions are identical, providing a powerful tool for $K$-sample testing.
A significant advantage of these statistics is their ability to operate in any Banach or metric space. By utilizing distance matrices, the methods can be applied to any data type where a distance metric is definable.
As shown in the implementation details, the statistics can handle numeric vectors, matrices, or data frames. The weight parameter allows users to choose between different forms (constant, probability, or chi-square), providing flexibility in balancing statistical power and computational requirements.
Feature Selection via BCor-SIS
For ultra-high dimensional datasets where $p \gg n$, the package implements BCor-SIS (Ball Correlation based Sure Independence Screening). This is a model-free feature selection procedure that ranks variables based on their Ball Correlation with a response variable. Theoretical results indicate that BCor-SIS can retain all truly relevant variables even when the number of dimensions grows exponentially relative to the sample size.
The screening process returns a list containing the indices of selected variables (ix) and complete statistical information (complete.info), which includes a matrix of weighted Ball Correlation statistics for each variable.
Datasets and Practical Applications
The versatility of the 'Ball' package is demonstrated through several built-in datasets, such as the ArcticLake sediment samples. These examples highlight the package's ability to handle 1D vectors, 2D matrices, and 3D arrays (like skull landmark coordinates) seamlessly. Furthermore, the inclusion of bd.gwas.test provides a specialized, high-performance implementation for Genome-Wide Association Studies, utilizing permutation and spectrum methods to manage the scale of genetic data.
Conclusion and Future Directions
The 'Ball' package offers a powerful, generic toolkit for detecting any form of dependence or distribution difference. Its geometric intuition—based on the proportion of data points within "closed balls"—allows it to handle complex structures without assuming specific functional forms. While the $O(n^2)$ time complexity presents a challenge for extremely large sample sizes, the implementation of multi-threading and specialized GWAS functions mitigates these costs for many practical applications.
Summary updated