| calculateMAType1Error {reproducer} | R Documentation |
calculateMAType1Error
Description
The function simulates multiple five group families of either two-group or four-group experiments and estimates the Type1 Error rate obtained after synthesizing the analysis results obtained from the experiments in each family. The Type1 Error is estimated as the percentage of families for which the overall mean of the five experiments was significantly different from zero. The experiment data may be one of four different type: Normal, Log-normal, Gamma or Laplace. The simulations can be repeated for different sample sizes depending on the parameter N. The output is a table of values identifying three observed effect size estimates (Cliff's d, PHat and StdMD) and their related type 1 error rates for each set of simulated families. The synthesis method for all three effect sizes is based on calculating the overall mean and variance for the family of experiments, and then using those values to calculate the effect size variance and its variance. This function supports the production of the values reported in data tables in the paper "Recommendations for Analyzing Small Sample Size Software Engineering Experiments" and its Supplementary Material.
Usage
calculateMAType1Error(
mean = 0,
sd = 1,
N = c(5, 10, 15, 20, 30, 40),
reps,
type = "n",
seed = 123,
Experiments = 5,
FourG = FALSE,
StdAdj = 0,
Blockmean = 0,
BlockStdAdj = 0,
StdExp = 0,
MAMethod = "PM",
alpha = 0.05
)
Arguments
mean |
This is the mean value of the control and treatment group(s) used in the simulations of each experiment of each family for simulations of a specified sample size (default 0). |
sd |
This is the standard deviation value of the control group(s) and treatment group(s) used in the simulations of each experiment of each family for simulations of a specified sample size (default 1). |
N |
This specifies the sample sizes per group that will be used in each set of simulations (default c(5,10,15,20,30,40)). |
reps |
The number of families simulated for each sample size. |
type |
This specifies the distribution of the data samples that will be simulated. Options ae "n" for Normal, "l", for Log-normal,'g" for Gamma, "lap" for LaPlace (default "n"). |
seed |
A seed for the simulations (default 123). |
Experiments |
The number of experiments in each family (default 5). |
FourG |
If FourG is FALSE the individual experiments in each family will be two-group experiments, otherwise the individual experiments will be four-group families (default FALSE). |
StdAdj |
Used to introduce variance heterogeneity for Laplace and Normal samples (default 0). |
Blockmean |
Used to set a fixed block effect for four-group experiments (default 0). |
BlockStdAdj |
Not used (default 0). |
StdExp |
Used to introduce heterogeneity among families of experiments (default 0). |
MAMethod |
Not used (default "PM"). |
alpha |
The significance level for statistical tests (default 0.05). |
Value
Design. Specifies the type of experiment 2G or 4G, the sample distribution (n,l,g,lap), and whether variance heterogeneity was added (het)
BEIncluded. Specifies whether or not a block effect was introduced. Always set to "No" for two-group experiments.
GrpSize. Specifies the size of each group in the individual experiments.
ObsPHat. The average of the average Phat value found for each family in the set of simulations.
ObsCliffd. The average of the average Cliffd value found for each family in the set of simulations.
ObsStdES. The average of StdMD calculated for each family in the set of simulations.
PHatType1ER. The percentage of the simulations, for a specific group size, for which the overall Phat estimate was significantly different from zero at the nominated alpha level.
CliffdType1ER. The percentage of the simulations, for a specific group size, for which the overall Cliff's d estimate was significantly different from zero at the nominated alpha level.
StdMDType1ER. The percentage of the simulations, for a specific group size, for which the overall StdMD estimate was significantly different from zero at the nominated alpha level.
Author(s)
Barbara Kitchenham and Lech Madeyski
Examples
# as.data.frame(calculateMAType1Error(mean=0,sd=1,N=c(5,10),reps=10,type="n",Experiments=5,
# FourG=FALSE,StdAdj=0,Blockmean=0,seed=123))
# Design BEIncluded GrpSize ObsPHat ObsCliffd ObsStdES PHatType1ER CliffdType1ER StdMDType1ER
#1 2G_n No 5 0.4848 -0.0304 -0.054156883 0 0 0.0
#2 2G_n No 10 0.5036 0.0072 0.002888142 0 0 0.1
#as.data.frame(calculateMAType1Error(mean=0,sd=1,N=c(5,10),reps=10,type="l",Experiments=5,
# FourG=FALSE,StdAdj=0,Blockmean=0,seed=123))
# Design BEIncluded GrpSize ObsPHat ObsCliffd ObsStdES PHatType1ER CliffdType1ER StdMDType1ER
#1 2G_l No 5 0.4848 -0.0304 -0.02789656 0 0 0.0
#2 2G_l No 10 0.5036 0.0072 0.06473696 0 0 0.2
#as.data.frame(calculateMAType1Error(mean=0,sd=1,N=c(5,10),reps=10,type="n",Experiments=5,
# FourG=TRUE,StdAdj=0.5,Blockmean=0.5,seed=123))
# Design BEIncluded GrpSize ObsPHat ObsCliffd ObsStdES PHatType1ER CliffdType1ER StdMDType1ER
# 1 4G_n_het Yes 5 0.5108 0.0216 0.01361820 0 0 0.1
# 2 4G_n_het Yes 10 0.5069 0.0138 0.01700672 0 0 0.0
as.data.frame(calculateMAType1Error(mean=0,sd=1,N=c(5,10),reps=5,type="l",Experiments=5,
FourG=TRUE,StdAdj=0,Blockmean=0.5,seed=123))
#Results for reps=10
# Design BEIncluded GrpSize ObsPHat ObsCliffd ObsStdES PHatType1ER CliffdType1ER StdMDType1ER
#1 4G_l Yes 5 0.5108 0.0216 0.07578257 0 0 0.2
#2 4G_l Yes 10 0.5072 0.0144 0.04839936 0 0 0.0