| proportion.test {LiftTest} | R Documentation |
A Bootstrap Proportion Test for Brand Lift Testing (Liu et al., 2023)
Description
This function implements several proportion tests that can be applied to Brand Lift Testing, including
-
\mathbf{clt}: Absolute lift based Z-test and relative lift based Z-test. The limiting distribution of Z-statistics are derived from the central limit theorem. -
\mathbf{bootstrap}: Absolute lift based bootstrap test (BS-A) and relative lift based bootstrap test (BS-R), see Liu et al., (2023). -
\mathbf{bootstrapmean}: Absolute lift based bootstrap mean test and relative lift based bootstrap mean test. (Efron and Tibshirani 1994). -
\mathbf{permutation}: Absolute lift based permutation test and relative lift based permutation test. (Efron and Tibshirani 1994).
Learn more about the proportion tests in the section Details.
Usage
proportion.test(data, method, B)
Arguments
data |
A 2x2 matrix with first column being (control sample size, treatment sample size) and the 2nd column being (control positive response count, treatment positive response count). |
method |
The method should be one of ("clt", "bootstrap", "bootstrapmean", "permutation") |
B |
Number of replications for bootstrap test or permutation test. Only required for methods "bootstrap", "bootstrapmean", "permutation". |
Details
\mathbf{clt}: the classic Z-test based on normal approximation. The absolute
lift based Z-test is defined as
Z = \frac{\hat p_1 - \hat p_0}{\sqrt{{s_0^2}/{n_0} + {s_1^2}/{n_1}}},
and he relative lift based Z-test is defined as
Z_r = \frac{\hat p_1 / \hat p_0 - 1}{\sqrt{s_1^2/(n_1\hat p_0^2) +
\hat p_1^2s_0^2/(n_0 \hat p_0^4)}},
where s_0^2 = \hat p_0(1-\hat p_0) and s_1^2 = \hat p_1(1-\hat p_1).
\mathbf{bootstrap}: the bootstrap proportion tests proposed in Liu et al., (2023),
see Algorithm 1 in their paper. There are two bootstrap tests: the absolute lift
based bootstrap test BS-A and the relative lift based bootstrap test BS-R. Note that
this type of bootstrap test is testing whether the distribution of the control group is
the same as the distribution of the treatment group. In the binomial distribution case,
it is equivalent to test whether the mean of the control group is
the same as the mean of the treatment group.
\mathbf{bootstrapmean} the bootstrap test to test whether the mean of the
control group is the same as the mean of the treatment group. See Algorithm 16.2
of Efron and Tibshirani (1994).
\mathbf{permutation} the permutation test to test whether the distribution of the
control group is the same as the distribution of the treatment group.
See Algorithm 15.1 of Efron and Tibshirani (1994).
Value
A list of absolute lift, relative lift, standardized absolute lift and their corresponding p-values. Standardized absolute lift equals absolute lift divided by its standard deviation. Only absolute lift and relative lift are available for method clt.
References
Wanjun Liu, Xiufan Yu, Jialiang Mao, Xiaoxu Wu, and Justin Dyer. 2023. Quantifying the Effectiveness of Advertising: A Bootstrap Proportion Test for Brand Lift Testing. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM ’23)
Efron, Bradley, and Robert J. Tibshirani. An introduction to the bootstrap. CRC press, 1994.
Examples
n1 <- 100; n2 <- 100; p1 <- 0.1; p2 <- 0.2
set.seed(1)
sim.data <- gen.simu.data(n1, n2, p1, p2, summary = TRUE)
result <- proportion.test(sim.data, method = "bootstrap", B = 1000)
relative.lift <- result$lift$relative
relative.lift.pval <- result$pvalue$relative