bestNormalize {bestNormalize}

Calculate and perform best normalizing transformation

Description

Performs a suite of normalizing transformations, and selects the best one on the basis of the Pearson P test statistic for normality. The transformation that has the lowest P (calculated on the transformed data) is selected. See details for more information.

Usage

bestNormalize(
  x,
  standardize = TRUE,
  allow_orderNorm = TRUE,
  allow_lambert_s = FALSE,
  allow_lambert_h = FALSE,
  allow_exp = TRUE,
  out_of_sample = TRUE,
  cluster = NULL,
  k = 10,
  r = 5,
  loo = FALSE,
  warn = FALSE,
  quiet = FALSE,
  tr_opts = list(),
  new_transforms = list(),
  norm_stat_fn = NULL,
  ...
)

## S3 method for class 'bestNormalize'
predict(object, newdata = NULL, inverse = FALSE, ...)

## S3 method for class 'bestNormalize'
print(x, ...)

## S3 method for class 'bestNormalize'
tidy(x, ...)

Arguments

x

A 'bestNormalize' object.

standardize

If TRUE, the transformed values are also centered and scaled, such that the transformation attempts a standard normal. This will not change the normality statistic.

allow_orderNorm

set to FALSE if orderNorm should not be applied

allow_lambert_s

Set to FALSE if the lambertW of type "s" should not be applied (see details). Expect about 2-3x elapsed computing time if TRUE.

allow_lambert_h

Set to TRUE if the lambertW of type "h" should be applied (see details). Expect about 2-3x elapsed computing time.

allow_exp

Set to TRUE if the exponential transformation should be applied (sometimes this will cause errors with heavy right skew)

out_of_sample

if FALSE, estimates quickly in-sample performance

cluster

name of cluster set using makeCluster

k

number of folds

r

number of repeats

loo

should leave-one-out CV be used instead of repeated CV? (see details)

warn

Should bestNormalize warn when a method doesn't work?

quiet

Should a progress-bar not be displayed for cross-validation progress?

tr_opts

a list (of lists), specifying options to be passed to each transformation (see details)

new_transforms

a named list of new transformation functions and their predict methods (see details)

norm_stat_fn

if specified, a function to calculate to assess normality (default is the Pearson chi-squared statistic divided by its d.f.)

...

not used

object

an object of class 'bestNormalize'

newdata

a vector of data to be (reverse) transformed

inverse

if TRUE, performs reverse transformation

Details

bestNormalize estimates the optimal normalizing transformation. This transformation can be performed on new data, and inverted, via the predict function.

This function currently estimates the Yeo-Johnson transformation, the Box Cox transformation (if the data is positive), the log_10(x+a) transformation, the square-root (x+a) transformation, and the arcsinh transformation. a is set to max(0, -min(x) + eps) by default. If allow_orderNorm == TRUE and if out_of_sample == FALSE then the ordered quantile normalization technique will likely be chosen since it essentially forces the data to follow a normal distribution. More information on the orderNorm technique can be found in the package vignette, or using ?orderNorm.

Repeated cross-validation is used by default to estimate the out-of-sample performance of each transformation if out_of_sample = TRUE. While this can take some time, users can speed it up by creating a cluster via the parallel package's makeCluster function, and passing the name of this cluster to bestNormalize via the cl argument. For best performance, we recommend the number of clusters to be set to the number of repeats r. Care should be taken to account for the number of observations per fold; too small a number and the estimated normality statistic could be inaccurate, or at least suffer from high variability.

As of version 1.3, users can use leave-one-out cross-validation as well for each method by setting loo to TRUE. This will take a lot of time for bigger vectors, but it will have the most accurate estimate of normalization efficacy. Note that if this method is selected, arguments k, r are ignored. This method will still work in parallel with the cl argument.

Note that the Lambert transformation of type "h" can be done by setting allow_lambert_h = TRUE, however this can take significantly longer to run.

Use tr_opts in order to set options for each transformation. For instance, if you want to overide the default a selection for log_x, set tr_opts$log_x = list(a = 1).

See the package's vignette on how to use custom functions with bestNormalize. All it takes is to create an S3 class and predict method for the new transformation and load it into the environment, then the new custom function (and its predict method) can be passed to bestNormalize with new_transform.

Value

A list of class bestNormalize with elements

x.t

transformed original data

x

original data

norm_stats

Pearson's Pearson's P / degrees of freedom

method

out-of-sample or in-sample, number of folds + repeats

chosen_transform

the chosen transformation (of appropriate class)

other_transforms

the other transformations (of appropriate class)

oos_preds

Out-of-sample predictions (if loo == TRUE) or normalization stats

The predict function returns the numeric value of the transformation performed on new data, and allows for the inverse transformation as well.

See Also

boxcox, orderNorm, yeojohnson

Examples

x <- rgamma(100, 1, 1)

## Not run: 
# With Repeated CV
BN_obj <- bestNormalize(x)
BN_obj
p <- predict(BN_obj)
x2 <- predict(BN_obj, newdata = p, inverse = TRUE)

all.equal(x2, x)

## End(Not run)


## Not run: 
# With leave-one-out CV
BN_obj <- bestNormalize(x, loo = TRUE)
BN_obj
p <- predict(BN_obj)
x2 <- predict(BN_obj, newdata = p, inverse = TRUE)

all.equal(x2, x)

## End(Not run)

# Without CV
BN_obj <- bestNormalize(x, allow_orderNorm = FALSE, out_of_sample = FALSE)
BN_obj
p <- predict(BN_obj)
x2 <- predict(BN_obj, newdata = p, inverse = TRUE)

all.equal(x2, x)