SLOPE {SLOPE} | R Documentation |
Sorted L-One Penalized Estimation
Description
Fit a generalized linear model regularized with the
sorted L1 norm, which applies a
non-increasing regularization sequence to the
coefficient vector () after having sorted it
in decreasing order according to its absolute values.
Usage
SLOPE(
x,
y,
family = c("gaussian", "binomial", "multinomial", "poisson"),
intercept = TRUE,
center = !inherits(x, "sparseMatrix"),
scale = c("l2", "l1", "sd", "none"),
alpha = c("path", "estimate"),
lambda = c("bh", "gaussian", "oscar", "lasso"),
alpha_min_ratio = if (NROW(x) < NCOL(x)) 0.01 else 1e-04,
path_length = if (alpha[1] == "estimate") 1 else 20,
q = 0.1 * min(1, NROW(x)/NCOL(x)),
theta1 = 1,
theta2 = 0.5,
prox_method = c("stack", "pava"),
screen = TRUE,
screen_alg = c("strong", "previous"),
tol_dev_change = 1e-05,
tol_dev_ratio = 0.995,
max_variables = NROW(x),
solver = c("fista", "admm"),
max_passes = 1e+06,
tol_abs = 1e-05,
tol_rel = 1e-04,
tol_rel_gap = 1e-05,
tol_infeas = 0.001,
tol_rel_coef_change = 0.001,
diagnostics = FALSE,
verbosity = 0,
sigma,
n_sigma,
lambda_min_ratio
)
Arguments
x |
the design matrix, which can be either a dense matrix of the standard matrix class, or a sparse matrix inheriting from Matrix::sparseMatrix. Data frames will be converted to matrices internally. |
y |
the response, which for |
family |
model family (objective); see Families for details. |
intercept |
whether to fit an intercept |
center |
whether to center predictors or not by their mean. Defaults
to |
scale |
type of scaling to apply to predictors.
|
alpha |
scale for regularization path: either a decreasing numeric vector (possibly of length 1) or a character vector; in the latter case, the choices are:
When a value is manually entered for |
lambda |
either a character vector indicating the method used to construct the lambda path or a numeric non-decreasing vector with length equal to the number of coefficients in the model; see section Regularization sequences for details. |
alpha_min_ratio |
smallest value for |
path_length |
length of regularization path; note that the path
returned may still be shorter due to the early termination criteria
given by |
q |
parameter controlling the shape of the lambda sequence, with
usage varying depending on the type of path used and has no effect
is a custom |
theta1 |
parameter controlling the shape of the lambda sequence
when |
theta2 |
parameter controlling the shape of the lambda sequence
when |
prox_method |
method for calculating the proximal operator for
the Sorted L1 Norm (the SLOPE penalty). Please see |
screen |
whether to use predictor screening rules (rules that allow some predictors to be discarded prior to fitting), which improve speed greatly when the number of predictors is larger than the number of observations. |
screen_alg |
what type of screening algorithm to use.
|
tol_dev_change |
the regularization path is stopped if the fractional change in deviance falls below this value; note that this is automatically set to 0 if a alpha is manually entered |
tol_dev_ratio |
the regularization path is stopped if the
deviance ratio |
max_variables |
criterion for stopping the path in terms of the maximum number of unique, nonzero coefficients in absolute value in model. For the multinomial family, this value will be multiplied internally with the number of levels of the response minus one. |
solver |
type of solver use, either |
max_passes |
maximum number of passes (outer iterations) for solver |
tol_abs |
absolute tolerance criterion for ADMM solver |
tol_rel |
relative tolerance criterion for ADMM solver |
tol_rel_gap |
stopping criterion for the duality gap; used only with FISTA solver. |
tol_infeas |
stopping criterion for the level of infeasibility; used with FISTA solver and KKT checks in screening algorithm. |
tol_rel_coef_change |
relative tolerance criterion for change in coefficients between iterations, which is reached when the maximum absolute change in any coefficient divided by the maximum absolute coefficient size is less than this value. |
diagnostics |
whether to save diagnostics from the solver (timings and other values depending on type of solver) |
verbosity |
level of verbosity for displaying output from the program. Setting this to 1 displays basic information on the path level, 2 a little bit more information on the path level, and 3 displays information from the solver. |
sigma |
deprecated; please use |
n_sigma |
deprecated; please use |
lambda_min_ratio |
deprecated; please use |
Details
SLOPE()
solves the convex minimization problem
where is a smooth and convex function and
the second part is the sorted L1-norm.
In ordinary least-squares regression,
is simply the squared norm of the least-squares residuals.
See section Families for specifics regarding the various types of
(model families) that are allowed in
SLOPE()
.
By default, SLOPE()
fits a path of models, each corresponding to
a separate regularization sequence, starting from
the null (intercept-only) model to an almost completely unregularized
model. These regularization sequences are parameterized using
and
, with only
varying along the
path. The length of the path can be manually, but will terminate
prematurely depending on
arguments
tol_dev_change
, tol_dev_ratio
, and max_variables
.
This means that unless these arguments are modified, the path is not
guaranteed to be of length path_length
.
Value
An object of class "SLOPE"
with the following slots:
coefficients |
a three-dimensional array of the coefficients from the model fit, including the intercept if it was fit. There is one row for each coefficient, one column for each target (dependent variable), and one slice for each penalty. |
nonzeros |
a three-dimensional logical array indicating whether a coefficient was zero or not |
lambda |
the lambda vector that when multiplied by a value in |
alpha |
vector giving the (unstandardized) scaling of the lambda sequence |
class_names |
a character vector giving the names of the classes for binomial and multinomial families |
passes |
the number of passes the solver took at each step on the path |
violations |
the number of violations of the screening rule at each step on the path;
only available if |
active_sets |
a list where each element indicates the indices of the coefficients that were active at that point in the regularization path |
unique |
the number of unique predictors (in absolute value) |
deviance_ratio |
the deviance ratio (as a fraction of 1) |
null_deviance |
the deviance of the null (intercept-only) model |
family |
the name of the family used in the model fit |
diagnostics |
a |
call |
the call used for fitting the model |
Families
Gaussian
The Gaussian model (Ordinary Least Squares) minimizes the following objective:
Binomial
The binomial model (logistic regression) has the following objective:
with .
Poisson
In poisson regression, we use the following objective:
Multinomial
In multinomial regression, we minimize the full-rank objective
with being the element in a
by
matrix, where
is the number of classes in the response.
Regularization Sequences
There are multiple ways of specifying the lambda
sequence
in SLOPE()
. It is, first of all, possible to select the sequence manually
by
using a non-increasing
numeric vector, possibly of length one, as argument instead of a character.
The greater the differences are between
consecutive values along the sequence, the more clustering behavior
will the model exhibit. Note, also, that the scale of the
vector makes no difference if
alpha = NULL
, since alpha
will be
selected automatically to ensure that the model is completely sparse at the
beginning and almost unregularized at the end. If, however, both
alpha
and lambda
are manually specified, then the scales of both do
matter, so make sure to choose them wisely.
Instead of choosing the sequence manually, one of the following automatically generated sequences may be chosen.
BH (Benjamini–Hochberg)
If lambda = "bh"
, the sequence used is that referred to
as by Bogdan et al, which sets
according to
for , where
is the quantile
function for the standard normal distribution and
is a parameter
that can be set by the user in the call to
SLOPE()
.
Gaussian
This penalty sequence is related to BH, such that
for , where
. We let
and
adjust the sequence to make sure that it's non-increasing.
Note that if
is large relative
to
, this option will result in a constant sequence, which is
usually not what you would want.
OSCAR
This sequence comes from Bondell and Reich and is a linear non-increasing sequence, such that
for . We use the parametrization from Zhong and Kwok
(2021) but use
and
instead of
and
to avoid confusion and abuse of notation.
lasso
SLOPE is exactly equivalent to the lasso when the sequence of regularization weights is constant, i.e.
for . Here, again, we stress that the fact that
all
are equal to one does not matter as long as
alpha == NULL
since we scale the vector automatically.
Note that this option is only here for academic interest and
to highlight the fact that SLOPE is
a generalization of the lasso. There are more efficient packages, such as
glmnet and biglasso, for fitting the lasso.
Solvers
There are currently two solvers available for SLOPE: FISTA (Beck and
Teboulle 2009) and ADMM (Boyd et al. 2008). FISTA is available for
families but ADMM is currently only available for family = "gaussian"
.
References
Bogdan, M., van den Berg, E., Sabatti, C., Su, W., & Candès, E. J. (2015). SLOPE – adaptive variable selection via convex optimization. The Annals of Applied Statistics, 9(3), 1103–1140.
Bondell, H. D., & Reich, B. J. (2008). Simultaneous Regression Shrinkage, Variable Selection, and Supervised Clustering of Predictors with OSCAR. Biometrics, 64(1), 115–123. JSTOR.
Boyd, S., Parikh, N., Chu, E., Peleato, B., & Eckstein, J. (2010). Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers. Foundations and Trends® in Machine Learning, 3(1), 1–122.
Beck, A., & Teboulle, M. (2009). A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems. SIAM Journal on Imaging Sciences, 2(1), 183–202.
See Also
plot.SLOPE()
, plotDiagnostics()
, score()
, predict.SLOPE()
,
trainSLOPE()
, coef.SLOPE()
, print.SLOPE()
, print.SLOPE()
,
deviance.SLOPE()
, sortedL1Prox()
Examples
# Gaussian response, default lambda sequence
fit <- SLOPE(bodyfat$x, bodyfat$y)
# Poisson response, OSCAR-type lambda sequence
fit <- SLOPE(
abalone$x,
abalone$y,
family = "poisson",
lambda = "oscar",
theta1 = 1,
theta2 = 0.9
)
# Multinomial response, custom alpha and lambda
m <- length(unique(wine$y)) - 1
p <- ncol(wine$x)
alpha <- 0.005
lambda <- exp(seq(log(2), log(1.8), length.out = p * m))
fit <- SLOPE(
wine$x,
wine$y,
family = "multinomial",
lambda = lambda,
alpha = alpha
)