bayesTest {bayesAB}R Documentation

Fit a Bayesian model to A/B test data.

Description

This function fits a Bayesian model to your A/B testing sample data. See Details for more information on usage.

Usage

bayesTest(
  A_data,
  B_data,
  priors,
  n_samples = 1e+05,
  distribution = c("bernoulli", "normal", "lognormal", "poisson", "exponential",
    "uniform", "bernoulliC", "poissonC")
)

Arguments

A_data

Vector of collected samples from recipe A

B_data

Vector of collected samples from recipe B

priors

Named vector or named list providing priors as required by the specified distribution:

  • For 'bernoulli' distribution list("alpha" = val1, "beta" = val2)

  • For 'normal' distribution c("mu" = val1, "lambda" = val2, "alpha" = val3, "beta" = val4)

  • For 'lognormal' distribution c("mu" = val1, "lambda" = val2, "alpha" = val3, "beta" = val4)

  • For 'poisson' distribution c("shape" = val1, "rate" = val2)

  • For 'exponential' distribution list("shape" = val1, "rate" = val2)

  • For 'uniform' distribution c("xm" = val1, "alpha" = val2)

  • For 'bernoulliC' distribution: same prior definitions as 'bernoulli'

  • For 'poissonC' distribution: same prior definitions as 'poisson'

See plotDistributions or the Note section of this help document for more info.

n_samples

Number of posterior samples to draw. Should be large enough for the distribution to converge. 1e5 is a good rule of thumb. Not used for closed form tests.

distribution

Distribution of underlying A/B test data.

Details

bayesTest is the main driver function of the bayesAB package. The input takes two vectors of data, corresponding to recipe A and recipe B of an A/B test. Order does not matter, except for interpretability of the final plots and intervals/point estimates. The Bayesian model for each distribution uses conjugate priors which must be specified at the time of invoking the function. Currently, there are eight supported distributions for the underlying data:

Value

A bayesTest object of the appropriate distribution class.

Note

For 'closed form' tests, you do not get a distribution over the posterior, but simply P(A > B) for the parameter in question.

Choosing priors correctly is very important. Please see http://fportman.com/writing/bayesab-0-dot-7-0-plus-a-primer-on-priors/ for a detailed example of choosing priors within bayesAB. Here are some ways to leverage objective/diffuse (assigning equal probability to all values) priors:

Keep in mind that the Prior Plots for bayesTest's run with diffuse priors may not plot correctly as they will not be truncated as they approach infinity. See plot.bayesTest for how to turn off the Prior Plots.

Examples

A_binom <- rbinom(100, 1, .5)
B_binom <- rbinom(100, 1, .6)

A_norm <- rnorm(100, 6, 1.5)
B_norm <- rnorm(100, 5, 2.5)

AB1 <- bayesTest(A_binom, B_binom, 
                 priors = c('alpha' = 1, 'beta' = 1), 
                 distribution = 'bernoulli')
AB2 <- bayesTest(A_norm, B_norm,
                 priors = c('mu' = 5, 'lambda' = 1, 'alpha' = 3, 'beta' = 1), 
                 distribution = 'normal')

print(AB1)
summary(AB1)
plot(AB1)

summary(AB2)

# Create a new variable that is the probability multiiplied
# by the normally distributed variable (expected value of something)

    AB3 <- combine(AB1, AB2, f = `*`, params = c('Probability', 'Mu'), newName = 'Expectation')

    print(AB3)
    summary(AB3)
    plot(AB3)


[Package bayesAB version 1.1.3 Index]