svmDP {DPpack} | R Documentation |

This class implements differentially private support vector machine (SVM) (Chaudhuri et al. 2011). It can be either weighted (Yang et al. 2005) or unweighted. Either the output or the objective perturbation method can be used for unweighted SVM, though only the output perturbation method is currently supported for weighted SVM.

To use this class for SVM, first use the `new`

method to
construct an object of this class with the desired function values and
hyperparameters, including a choice of the desired kernel. After
constructing the object, the `fit`

method can be applied to fit the
model with a provided dataset, data bounds, and optional observation
weights and weight upper bound. In fitting, the model stores a vector of
coefficients `coeff`

which satisfy differential privacy. Additionally,
if a nonlinear kernel is chosen, the models stores a mapping function from
the input data X to a higher dimensional embedding V in the form of a
method `XtoV`

as required (Chaudhuri et al. 2011). These
can be released directly, or used in conjunction with the `predict`

method to privately predict the label of new datapoints. Note that the
mapping function `XtoV`

is based on an approximation method via
Fourier transforms (Rahimi and Recht 2007; Rahimi and Recht 2008).

Note that in order to guarantee differential privacy for the SVM model, certain constraints must be satisfied for the values used to construct the object, as well as for the data used to fit. These conditions depend on the chosen perturbation method. First, the loss function is assumed to be differentiable (and doubly differentiable if the objective perturbation method is used). The hinge loss, which is typically used for SVM, is not differentiable at 1. Thus, to satisfy this constraint, this class utilizes the Huber loss, a smooth approximation to the hinge loss (Chapelle 2007). The level of approximation to the hinge loss is determined by a user-specified constant, h, which defaults to 0.5, a typical value. Additionally, the regularizer must be 1-strongly convex and differentiable. It also must be doubly differentiable if objective perturbation is chosen. If weighted SVM is desired, the provided weights must be nonnegative and bounded above by a global or public value, which must also be provided.

Finally, it is assumed that if x represents a single row of the dataset X,
then the l2-norm of x is at most 1 for all x. In order to ensure this
constraint is satisfied, the dataset is preprocessed and scaled, and the
resulting coefficients are postprocessed and un-scaled so that the stored
coefficients correspond to the original data. Due to this constraint on x,
it is best to avoid using a bias term in the model whenever possible. If a
bias term must be used, the issue can be partially circumvented by adding a
constant column to X before fitting the model, which will be scaled along
with the rest of X. The `fit`

method contains functionality to add a
column of constant 1s to X before scaling, if desired.

`DPpack::EmpiricalRiskMinimizationDP.CMS`

-> `DPpack::WeightedERMDP.CMS`

-> `svmDP`

`new()`

Create a new `svmDP`

object.

svmDP$new( regularizer, eps, gamma, perturbation.method = "objective", kernel = "linear", D = NULL, kernel.param = NULL, regularizer.gr = NULL, huber.h = 0.5 )

`regularizer`

String or regularization function. If a string, must be 'l2', indicating to use l2 regularization. If a function, must have form

`regularizer(coeff)`

, where`coeff`

is a vector or matrix, and return the value of the regularizer at`coeff`

. See`regularizer.l2`

for an example. Additionally, in order to ensure differential privacy, the function must be 1-strongly convex and doubly differentiable.`eps`

Positive real number defining the epsilon privacy budget. If set to Inf, runs algorithm without differential privacy.

`gamma`

Nonnegative real number representing the regularization constant.

`perturbation.method`

String indicating whether to use the 'output' or the 'objective' perturbation methods (Chaudhuri et al. 2011). Defaults to 'objective'.

`kernel`

String indicating which kernel to use for SVM. Must be one of 'linear', 'Gaussian'. If 'linear' (default), linear SVM is used. If 'Gaussian,' uses the sampling function corresponding to the Gaussian (radial) kernel approximation.

`D`

Nonnegative integer indicating the dimensionality of the transform space approximating the kernel if a nonlinear kernel is used. Higher values of D provide better kernel approximations at a cost of computational efficiency. This value must be specified if a nonlinear kernel is used.

`kernel.param`

Positive real number corresponding to the Gaussian kernel parameter. Defaults to 1/p, where p is the number of predictors.

`regularizer.gr`

Optional function representing the gradient of the regularization function with respect to

`coeff`

and of the form`regularizer.gr(coeff)`

. Should return a vector. See`regularizer.gr.l2`

for an example. If`regularizer`

is given as a string, this value is ignored. If not given and`regularizer`

is a function, non-gradient based optimization methods are used to compute the coefficient values in fitting the model.`huber.h`

Positive real number indicating the degree to which the Huber loss approximates the hinge loss. Defaults to 0.5 (Chapelle 2007).

A new svmDP object.

`fit()`

Fit the differentially private SVM model. This method runs
either the output perturbation or the objective perturbation algorithm
(Chaudhuri et al. 2011), depending on the value of
perturbation.method used to construct the object, to generate an
objective function. A numerical optimization method is then run to find
optimal coefficients for fitting the model given the training data,
weights, and hyperparameters. The built-in `optim`

function
using the "BFGS" optimization method is used. If `regularizer`

is
given as 'l2' or if `regularizer.gr`

is given in the construction of
the object, the gradient of the objective function is utilized by
`optim`

as well. Otherwise, non-gradient based optimization methods
are used. The resulting privacy-preserving coefficients are stored in
`coeff`

.

svmDP$fit( X, y, upper.bounds, lower.bounds, add.bias = FALSE, weights = NULL, weights.upper.bound = NULL )

`X`

Dataframe of data to be fit.

`y`

Vector or matrix of true labels for each row of

`X`

.`upper.bounds`

Numeric vector of length

`ncol(X)`

giving upper bounds on the values in each column of X. The`ncol(X)`

values are assumed to be in the same order as the corresponding columns of`X`

. Any value in the columns of`X`

larger than the corresponding upper bound is clipped at the bound.`lower.bounds`

Numeric vector of length

`ncol(X)`

giving lower bounds on the values in each column of`X`

. The`ncol(X)`

values are assumed to be in the same order as the corresponding columns of`X`

. Any value in the columns of`X`

larger than the corresponding upper bound is clipped at the bound.`add.bias`

Boolean indicating whether to add a bias term to

`X`

. Defaults to FALSE.`weights`

Numeric vector of observation weights of the same length as

`y`

. If not given, no observation weighting is performed.`weights.upper.bound`

Numeric value representing the global or public upper bound on the weights. Required if weights are given.

`XtoV()`

Convert input data X into transformed data V. Uses sampled pre-filter values and a mapping function based on the chosen kernel to produce D-dimensional data V on which to train the model or predict future values. This method is only used if the kernel is nonlinear. See Chaudhuri et al. (2011) for more details.

svmDP$XtoV(X)

`X`

Matrix corresponding to the original dataset.

Matrix V of size n by D representing the transformed dataset, where n is the number of rows of X, and D is the provided transformed space dimension.

`predict()`

Predict label(s) for given `X`

using the fitted
coefficients.

svmDP$predict(X, add.bias = FALSE, raw.value = FALSE)

`X`

Dataframe of data on which to make predictions. Must be of same form as

`X`

used to fit coefficients.`add.bias`

Boolean indicating whether to add a bias term to

`X`

. Defaults to FALSE. If add.bias was set to TRUE when fitting the coefficients, add.bias should be set to TRUE for predictions.`raw.value`

Boolean indicating whether to return the raw predicted value or the rounded class label. If FALSE (default), outputs the predicted labels 0 or 1. If TRUE, returns the raw score from the SVM model.

Matrix of predicted labels or scores corresponding to each row of
`X`

.

`clone()`

The objects of this class are cloneable with this method.

svmDP$clone(deep = FALSE)

`deep`

Whether to make a deep clone.

Chaudhuri K, Monteleoni C, Sarwate AD (2011).
“Differentially Private Empirical Risk Minimization.”
*Journal of Machine Learning Research*, **12**(29), 1069-1109.
https://jmlr.org/papers/v12/chaudhuri11a.html.

Yang X, Song Q, Cao A (2005).
“Weighted support vector machine for data classification.”
In *Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005.*, volume 2, 859-864 vol. 2.
doi:10.1109/IJCNN.2005.1555965.

Chapelle O (2007).
“Training a Support Vector Machine in the Primal.”
*Neural Computation*, **19**(5), 1155-1178.
doi:10.1162/neco.2007.19.5.1155.

Rahimi A, Recht B (2007).
“Random Features for Large-Scale Kernel Machines.”
In Platt J, Koller D, Singer Y, Roweis S (eds.), *Advances in Neural Information Processing Systems*, volume 20.
https://proceedings.neurips.cc/paper/2007/file/013a006f03dbc5392effeb8f18fda755-Paper.pdf.

Rahimi A, Recht B (2008).
“Weighted Sums of Random Kitchen Sinks: Replacing minimization with randomization in learning.”
In Koller D, Schuurmans D, Bengio Y, Bottou L (eds.), *Advances in Neural Information Processing Systems*, volume 21.
https://proceedings.neurips.cc/paper/2008/file/0efe32849d230d7f53049ddc4a4b0c60-Paper.pdf.

```
# Build train dataset X and y, and test dataset Xtest and ytest
N <- 400
X <- data.frame()
y <- data.frame()
for (i in (1:N)){
Xtemp <- data.frame(x1 = stats::rnorm(1,sd=.28) , x2 = stats::rnorm(1,sd=.28))
if (sum(Xtemp^2)<.15) ytemp <- data.frame(y=0)
else ytemp <- data.frame(y=1)
X <- rbind(X, Xtemp)
y <- rbind(y, ytemp)
}
Xtest <- X[seq(1,N,10),]
ytest <- y[seq(1,N,10),,drop=FALSE]
X <- X[-seq(1,N,10),]
y <- y[-seq(1,N,10),,drop=FALSE]
# Construct object for SVM
regularizer <- 'l2' # Alternatively, function(coeff) coeff%*%coeff/2
eps <- 1
gamma <- 1
perturbation.method <- 'output'
kernel <- 'Gaussian'
D <- 20
svmdp <- svmDP$new(regularizer, eps, gamma, perturbation.method,
kernel=kernel, D=D)
# Fit with data
# Bounds for X based on construction
upper.bounds <- c( 1, 1)
lower.bounds <- c(-1,-1)
weights <- rep(1, nrow(y)) # Uniform weighting
weights[nrow(y)] <- 0.5 # Half weight for last observation
wub <- 1 # Public upper bound for weights
svmdp$fit(X, y, upper.bounds, lower.bounds, weights=weights,
weights.upper.bound=wub) # No bias term
# Predict new data points
predicted.y <- svmdp$predict(Xtest)
n.errors <- sum(predicted.y!=ytest)
```

[Package *DPpack* version 0.1.0 Index]