| significance_analysis {bandit} | R Documentation |
significance_analysis
Description
A convenience function to perform overall proportion comparison using prop.test, before doing pairwise comparisons, to see what outcomes seem to be better than others.
Usage
significance_analysis(x, n)
Arguments
x |
as in prop.test, a vector of the number of successes |
n |
as in prop.test, a vector of the number of trials |
Value
a data frame with the following columns:
successes |
x |
totals |
n |
estimated_proportion |
x/n |
lower |
0.95 confidence interval on the estimated amount by which this alternative outperforms the next-lower alternative |
upper |
0.95 confidence interval on the estimated amount by which this alternative outperforms the next-lower alternative |
significance |
p-value for the test that this alternative outperforms the next-lower alternative |
order |
order, by highest success proportion |
best |
1 if it is part of the 'highest performing group' – those groups which were not significantly different from the best group |
p_best |
Bayesian posterior probability that this alternative is the best binomial bandit |
Note
This is intended for use in A/B split testing – so sizes of n should be roughly equal. Also, note that alternatives which have the same rank are grouped together for analysis with the 'next-lower' alternative, so you may want to check to see if ranks are equal.
Author(s)
Thomas Lotze <thomaslotze@thomaslotze.com>
See Also
Examples
x = c(10,20,30,50)
n = c(100,102,120,130)
sa = significance_analysis(x,n)
sa[rev(order(sa$estimated_proportion)), ]
x = c(37,41,30,43,39,30,31,35,50,30)
n = rep(50, length(x))
sa = significance_analysis(x,n)
sa[rev(order(sa$estimated_proportion)), ]
x = c(37,41,30,43,39,30,31,37,50,30)
n = rep(50, length(x))
sa = significance_analysis(x,n)
sa[rev(order(sa$estimated_proportion)), ]