Highly Adaptive Ridge

arXiv:2410.02680v2 Announce Type: replace-cross Abstract: In this paper we propose the Highly Adaptive Ridge (HAR): a regression method that achieves an $n^{-1/3}$ dimension-free $\mathscr L_2$ convergence rate (up to log factors) in the class of right-continuous functions with square-integrable sectional derivatives. This is a large nonparametric function class that is particularly appropriate for tabular data. For squared error loss, HAR is exactly kernel ridge regression with a specific data-adaptive kernel based on a saturated zero-order tensor-product spline basis expansion, and thus scales reasonably well, computationally. We prove close connections between HAR and kernel ridge regression in Sobolev spaces of dominating mixed derivatives, and highlight practical differences. We use simulation and real data to confirm our theory.