significance_analysis {bandit}R Documentation

significance_analysis

Description

A convenience function to perform overall proportion comparison using prop.test, before doing pairwise comparisons, to see what outcomes seem to be better than others.

Usage

significance_analysis(x, n)

Arguments

x

as in prop.test, a vector of the number of successes

n

as in prop.test, a vector of the number of trials

Value

a data frame with the following columns:

successes

x

totals

n

estimated_proportion

x/n

lower

0.95 confidence interval on the estimated amount by which this alternative outperforms the next-lower alternative

upper

0.95 confidence interval on the estimated amount by which this alternative outperforms the next-lower alternative

significance

p-value for the test that this alternative outperforms the next-lower alternative

order

order, by highest success proportion

best

1 if it is part of the 'highest performing group' – those groups which were not significantly different from the best group

p_best

Bayesian posterior probability that this alternative is the best binomial bandit

Note

This is intended for use in A/B split testing – so sizes of n should be roughly equal. Also, note that alternatives which have the same rank are grouped together for analysis with the 'next-lower' alternative, so you may want to check to see if ranks are equal.

Author(s)

Thomas Lotze <thomaslotze@thomaslotze.com>

See Also

prop.test

Examples

x = c(10,20,30,50)
n = c(100,102,120,130)
sa = significance_analysis(x,n)
sa[rev(order(sa$estimated_proportion)), ]

x = c(37,41,30,43,39,30,31,35,50,30)
n = rep(50, length(x))
sa = significance_analysis(x,n)
sa[rev(order(sa$estimated_proportion)), ]

x = c(37,41,30,43,39,30,31,37,50,30)
n = rep(50, length(x))
sa = significance_analysis(x,n)
sa[rev(order(sa$estimated_proportion)), ]

[Package bandit version 0.5.0 Index]