Introduction: The Challenge of Categorical Data
Categorical data is ubiquitous in modern datasets, yet it remains a significant hurdle for traditional statistical analysis. Because categories like "Apple," "Orange," or "Banana" exist on a nominal scale, standard operations like calculating mean, variance, or covariance are mathematically undefined.
To bridge this gap, researchers typically resort to data encoding. However, arbitrary label assignments (e.g., Apple=1, Orange=2) can introduce spurious correlations, while techniques like "one-hot encoding" drastically increase dimensionality without adding intrinsic information. Existing tools like Mutual Information (MI) or Distance Correlation (dCor) often struggle with multi-level categorical targets or exhibit sensitivity to label permutations. The Categorical Neighbour Correlation Coefficient (CnCor) addresses these issues by providing a "label-blind" measure capable of capturing intricate, non-linear dependencies directly from categorical data.
The CnCor Framework: A Label-Blind Approach
CnCor builds upon the neighbour correlation coefficient (nCor), which assesses functional relationships by determining if points close in the feature space share similar target values. The authors adapt this for categorical variables through three primary innovations:
- Indicator-Based Distance: Instead of Euclidean metrics, CnCor utilizes indicator functions to determine categorical similarity. A normalization scheme is introduced to allow for the seamless comparison of categorical and quantitative features on a unified scale.
- Reformulated Correlation: By replacing standard product-moment covariance with an indicator function that checks for category matches among neighboring points, the measure becomes entirely insensitive to specific numeric labels. It cares only if categories are identical or distinct.
- Statistical Significance: The raw score is rescaled to a consistent 0–1 range, where 0 denotes independence and 1 indicates a perfect functional relationship. This is paired with a derived theoretical threshold for statistical significance.
To enhance practical utility, the authors also introduced a Fast Distance Updating Strategy for the Coefficient of Essentialness (COE). This allows for efficient backward elimination feature selection (COE-BES) by updating distance matrices rather than recomputing them, significantly reducing computational overhead.
Experimental Validation
The authors evaluated CnCor using five synthetic datasets (E1–E5) designed to test various relationship types, including trigonometric functions, product-based interactions, and complex non-linear effects with noise.
A key finding from these synthetic tests is label insensitivity. While traditional measures like nCor fluctuate wildly based on arbitrary labeling, CnCor remains perfectly stable. Furthermore, in complex interaction scenarios (Experiment E5), where individual features appear independent of the target but are predictive in combination, CnCor clearly identified the relationship while most baseline scores (including HSIC, dCor, and RDC) fell below significance thresholds.
Performance in Feature Selection
Beyond synthetic tests, CnCor was benchmarked against 11 established feature selection methods across 9 real-world datasets, including Madelon, Isolet, and Arrhythmia. These datasets represent diverse challenges in image recognition, medical diagnosis, and high-dimensional signal processing.
Using the COE-BES algorithm, the authors achieved higher classification accuracy with fewer features compared to state-of-the-art baselines. The results demonstrate that CnCor's ability to capture local functional relationships translates into superior feature subsets for downstream machine learning tasks like k-NN and AdaBoost.
Conclusion and Future Directions
CnCor provides a robust, mathematically grounded framework for analyzing categorical dependencies without the pitfalls of traditional encoding. Its strengths lie in its label insensitivity and its capacity to detect high-order interactions that global measures often miss.
While the fast update strategy makes backward elimination feasible, future work may focus on scaling the underlying nearest-neighbor searches for datasets with millions of observations. Additionally, further refinement in handling mixed targets—those containing both continuous and discrete values—could expand the applicability of CnCor in complex real-world environments.
Summary updated