Comparative Evaluation of Hyperparameter Optimization Strategies for k-Nearest Neighbors over Mixed Domain Search Spaces

Abstract

k-Nearest Neighbors (KNN) remains a strong baseline due to its simplicity, interpretability, and non-parametric structure; however, its predictive performance is highly sensitive to interacting hyperparameters. This paper studies search-constrained hyperparameter optimization for KNN over a mixed search space comprising an integer neighborhood size, a categorical distance metric, and a continuous Minkowski exponent. Seven tuning strategies are compared under a unified nested validation protocol with standardized preprocessing: grid search, random search, Bayesian optimization, genetic algorithm, surrogate optimization, particle swarm optimization, and grey wolf optimizer. Experiments on diverse public classification datasets from UCI, OpenML, and Kaggle evaluate predictive performance (accuracy, macro-AUC, cross-entropy), validation loss, and runtime. The results indicate that adaptive optimizers generally provide more favorable performance–cost trade-offs than exhaustive grids under limited evaluations, while no single method is uniformly best across all datasets. Rank-based statistical comparisons using non-parametric tests further support the observed differences and motivate practical recommendations for selecting tuning strategies as a function of evaluation search and mixed-domain search-space characteristics.