calculate4GType1Error {reproducer}

calculate4GType1Error

Description

The function simulates multiple four-group experiments and estimates the Type1 Error rate obtained from the set of simulated experiments. The Type1 Error is estimated as the percentage of experiments for which the mean the experiment was significantly different from zero at the 0.05 significance level using two-sided tests. The experiment data may be one of four different type: Normal, Log-normal, Gamma or Laplace. The output is a set of values identifying three observed effect size estimates (Cliff's d, PHat and StdMD) and their related type 1 error rates. This function supports the production of the values reported in data tables in the paper "Recommendations for Analyzing Small Sample Size Software Engineering Experiments" and its Supplementary Material.

Usage

calculate4GType1Error(
  mean = 0,
  sd = 1,
  N = 10,
  reps = 10,
  type = "n",
  seed = 123,
  StdAdj = 0,
  Blockmean = 0
)

Arguments

mean

This is the mean value of the control and treatment group(s) used in the simulations (default 0).

sd

This is the standard deviation value of the control group(s) and treatment group(s) used in the simulations (default 1).

N

This specifies the sample size per group that will be used in each simulation (default 5).

reps

The number of experiments to simulated.

type

This specifies the distribution of the data samples that will be simulated. Options are "n" for Normal, "l", for Log-normal,'g" for Gamma, "lap" for LaPlace (default "n").

seed

A seed for the simulations (default 123).

StdAdj

Used to introduce variance heterogeneity for Laplace and Normal samples (default 0).

Blockmean

Used to specify the block effect (default 0).

Value

Design. Specifies the type of experiment 2G or 4G, the sample distribution (n,l,g,lap), and whether variance heterogeneity was added (het)

GrpSize. Specifies the size of each group in the simulations.

BEIncluded. Specifies whether or not a block effect was introduced.

ObsPHat. The average of the average Phat values found in the set of simulations.

ObsCliffd. The average of the average Cliffd values found in the set of simulations.

ObsStdES. The average of StdMD values found in the set of simulations.

PHatType1ER. The percentage of the simulations for which the Phat estimate was significantly different from zero at the 0.05 alpha level.

CliffdType1ER. The percentage of the simulations for which the overall Cliff's d estimate was significantly different from zero at the 0.05 alpha level.

StdMDType1ER. The percentage of the simulations for which the overall StdMD estimate was significantly different from zero at the 0.05 significance level.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

as.data.frame(calculate4GType1Error(mean=0,sd=1,N=40,reps=100,type="n",seed=17+1056,StdAdj = 0.5,
 Blockmean=0.5))
 # Results for reps=300
#    Design GrpSize BEIncluded   ObsPHat   ObsCliffd   ObsStdES PHatType1ER CliffdType1ER StdES..
#1 4G_n_het      40        Yes 0.5034729 0.006945833 0.01316457        0.03    0.02333333 0.046..

#as.data.frame(calculate4GType1Error(mean=0,sd=1,N=40,reps=300,type="lap",seed=17+2056,
#  StdAdj = 0.5,Blockmean=0.5))
#      Design GrpSize BEIncluded   ObsPHat    ObsCliffd  ObsStdES PHatType1ER CliffdType1ER Std..
#1 4G_lap_het      40        Yes 0.4992708 -0.001458333 0.0014446  0.04333333          0.04  0.06