bayesTest {bayesAB}  R Documentation 
Fit a Bayesian model to A/B test data.
Description
This function fits a Bayesian model to your A/B testing sample data. See Details for more information on usage.
Usage
bayesTest(
A_data,
B_data,
priors,
n_samples = 1e+05,
distribution = c("bernoulli", "normal", "lognormal", "poisson", "exponential",
"uniform", "bernoulliC", "poissonC")
)
Arguments
A_data 
Vector of collected samples from recipe A 
B_data 
Vector of collected samples from recipe B 
priors 
Named vector or named list providing priors as required by the specified distribution:
See plotDistributions or the Note section of this help document for more info. 
n_samples 
Number of posterior samples to draw. Should be large enough for the distribution to converge. 1e5 is a good rule of thumb. Not used for closed form tests. 
distribution 
Distribution of underlying A/B test data. 
Details
bayesTest
is the main driver function of the bayesAB package. The input takes two vectors of data,
corresponding to recipe A and recipe B of an A/B test. Order does not matter, except for interpretability of the final
plots and intervals/point estimates. The Bayesian model for each distribution uses conjugate priors which must
be specified at the time of invoking the function. Currently, there are eight supported distributions for the underlying data:
Bernoulli: If your data is well modeled by 1s and 0s, according to a specific probability
p
of a 1 occurringFor example, clickthroughrate /conversions for a page
Data must be in a {0, 1} format where 1 corresponds to a 'success' as per the Bernoulli distribution
Uses a conjugate
Beta
distribution for the parameter p in the Bernoulli distribution
alpha
andbeta
must be set for a prior distribution over palpha = 1, beta = 1 can be used as a diffuse or uniform prior
Normal: If your data is well modeled by the normal distribution, with parameters
\mu
,\sigma^2
controlling mean and variance of the underlying distributionData can be negative if it makes sense for your experiment
Uses a conjugate
NormalInverseGamma
distribution for the parameters\mu
and\sigma^2
in the Normal Distribution.
mu
,lambda
,alpha
, andbeta
must be set for prior distributions over\mu, \sigma^2
in accordance with the parameters of the conjugate prior distributions:
\mu, \sigma^2
~ NormalInverseGamma(mu, lambda, alpha, beta)

This is a bivariate distribution (commonly used to model mean and variance of the normal distribution). You may want to experiment with both this distribution and the
plotNormal
andplotInvGamma
outputs separately before arriving at a suitable set of priors for the Normal and LogNormalbayesTest
.
LogNormal: If your data is well modeled by the lognormal distribution, with parameters
\mu
,\sigma^2
as the parameters of the corresponding lognormal distribution (log of data is ~ N(\mu
,\sigma^2
))Support for a lognormal distribution is strictly positive
The Bayesian model requires same conjugate priors on
\mu
,\sigma^2
as for the Normal Distribution priorsNote: The
\mu
and\sigma^2
are not the mean/variance of lognormal numbers themselves but are rather the corresponding parameters of the lognormal distribution. Thus, posteriors for the statistics 'Mean' and 'Variance' are returned alongside 'Mu' and 'Sig_Sq' for interpretability.
Poisson: If your data is well modeled by the Poisson distribution, with parameter
\lambda
controlling the average number of events per interval.For example, pageviews per session
Data must be strictly integral or 0.
Uses a conjugate
Gamma
distribution for the parameter\lambda
in the Poisson Distribution
shape
andrate
must be set for prior distribution over\lambda
Exponential: If your data is well modeled by the Exponential distribution, with parameter
\lambda
controlling the rate of decay.For example, time spent on a page or customers' LTV
Data must be strictly >= 0
Uses a conjugate
Gamma
distribution for the parameter\lambda
in the Exponential Distribution
shape
andrate
must be set for prior distribution over\lambda
Uniform: If your data is well modeled by the Uniform distribution, with parameter
\theta
controlling the max value.bayesAB has only implemented Uniform(0,
\theta
) formsFor example, estimating max/total inventory size from individually numbered snapshots
Data must be strictly > 0
Uses a conjugate
Pareto
distribution for the parameter\theta
in the Uniform(0,\theta
) Distribution
xm
andalpha
must be set for prior distribution over\theta
BernoulliC: Closed form (computational) calculation of the 'bernoulli' bayesTest. Same priors are required.
PoissonC: Closed form (computational) calculation of the 'poisson' bayesTest. Same priors are required.
Value
A bayesTest
object of the appropriate distribution class.
Note
For 'closed form' tests, you do not get a distribution over the posterior, but simply P(A > B) for the parameter in question.
Choosing priors correctly is very important. Please see http://fportman.com/writing/bayesab0dot70plusaprimeronpriors/ for a detailed example of choosing priors within bayesAB. Here are some ways to leverage objective/diffuse (assigning equal probability to all values) priors:

Beta
(1, 1) 
Gamma
(eps, eps) ~Gamma
(.00005, .00005) will be effectively diffuse 
InvGamma
(eps, eps) ~InvGamma
(.00005, .00005) will be effectively diffuse 
Pareto
(eps, eps) ~Pareto
(.005, .005) will be effectively diffuse
Keep in mind that the Prior Plots for bayesTest's run with diffuse priors may not plot correctly as they will not be truncated as they approach infinity. See plot.bayesTest for how to turn off the Prior Plots.
Examples
A_binom < rbinom(100, 1, .5)
B_binom < rbinom(100, 1, .6)
A_norm < rnorm(100, 6, 1.5)
B_norm < rnorm(100, 5, 2.5)
AB1 < bayesTest(A_binom, B_binom,
priors = c('alpha' = 1, 'beta' = 1),
distribution = 'bernoulli')
AB2 < bayesTest(A_norm, B_norm,
priors = c('mu' = 5, 'lambda' = 1, 'alpha' = 3, 'beta' = 1),
distribution = 'normal')
print(AB1)
summary(AB1)
plot(AB1)
summary(AB2)
# Create a new variable that is the probability multiiplied
# by the normally distributed variable (expected value of something)
AB3 < combine(AB1, AB2, f = `*`, params = c('Probability', 'Mu'), newName = 'Expectation')
print(AB3)
summary(AB3)
plot(AB3)