about
Fractional Ridge Regression (arxiv.org)
1 point by mwakanosya on May 13, 2020 | hide | past | pdf | discuss on HN

In plain words: Instead of tuning how strongly ridge regression shrinks coefficients, this sets the shrunken coefficients' size relative to the unshrunk ones, so settings are easy to compare across datasets. Each setting gives a distinct answer and covers the useful range automatically, and stays fast at scale.

Abstract · Fractional ridge regression: a fast, interpretable reparameterization of ridge regression

Ridge regression (RR) is a regularization technique that penalizes the L2-norm of the coefficients in linear regression. One of the challenges of using RR is the need to set a hyperparameter ($α$) that controls the amount of regularization. Cross-validation is typically used to select the best $α$ from a set of candidates. However, efficient and appropriate selection of $α$ can be challenging, particularly where large amounts of data are analyzed. Because the selected $α$ depends on the scale of the data and predictors, it is not straightforwardly interpretable. Here, we propose to reparameterize RR in terms of the ratio $γ$ between the L2-norms of the regularized and unregularized coefficients. This approach, called fractional RR (FRR), has several benefits: the solutions obtained for different $γ$ are guaranteed to vary, guarding against wasted calculations, and automatically span the relevant range of regularization, avoiding the need for arduous manual exploration. We provide an algorithm to solve FRR, as well as open-source software implementations in Python and MATLAB (https://github.com/nrdg/fracridge). We show that the proposed method is fast and scalable for large-scale data problems, and delivers results that are straightforward to interpret and compare across models and datasets.

Ariel Rokem, Kendrick Kay
arXiv:2005.03220 · stat.ME, cs.LG, stat.ML · submitted May 7, 2020
abstract · pdf · html

add comment on HN