calculate2GType1Error {reproducer}

calculate2GType1Error

Description

The function simulates multiple two-group experiments and estimates the Type1 Error rate obtained from the set of simulated experiments. The Type1 Error is estimated as the percentage of experiments for which the mean the experiment was significantly different from zero at the 0.05 significance level using two-sided tests. The experiment data may be one of four different type: Normal, Log-normal, Gamma or Laplace. The output is a set of values identifying three observed effect size estimates (Cliff's d, PHat and StdMD) and their related type 1 error rates. This function supports the production of the values reported in data tables in the paper "Recommendations for Analyzing Small Sample Size Software Engineering Experiments" and its Supplementary Material.

Usage

calculate2GType1Error(
  mean = 0,
  sd = 1,
  N = 10,
  reps,
  type = "n",
  seed = 123,
  StdAdj = 0
)

Arguments

mean

This is the mean value of the control and treatment group(s) used in the simulations of each experiment for simulations of a specified sample size (default 0).

sd

This is the standard deviation value of the control group(s) and treatment group(s) used in the simulations of each experiment of each family for simulations of a specified sample size (default 1).

N

This specifies the sample size per group that will be used in each set of simulations (default 5).

reps

The number of experiments to simulated.

type

This specifies the distribution of the data samples that will be simulated. Options ae "n" for Normal, "l", for Log-normal,'g" for Gamma, "lap" for LaPlace (default "n").

seed

A seed for the simulations (default 123).

StdAdj

Used to introduce variance heterogeneity for Laplace and Normal samples(default 0).

Value

Design. Specifies the type of experiment 2G or 4G, the sample distribution (n,l,g,lap), and whether variance heterogeneity was added (het)

GrpSize. Specifies the size of each group in the individual experiments.

ObsPHat. The average Phat values found in the set of simulations.

ObsCliffd. The average Cliffd values found in the set of simulations.

ObsStdES. The average of StdMD values found in the set of simulations.

PHatType1ER. The proportion of the simulations for which the Phat estimate was significantly different from zero at the nominated alpha level.

CliffdType1ER. The proportion of the simulations for which the Cliff's d estimate was significantly different from zero at the nominated alpha level.

StdMDType1ER. The proportion of the simulations for which the StdMD estimate was significantly different from zero at the nominated 0.05 significance level.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

calculate2GType1Error(mean=1,sd=3,N=10,reps=100,type="g",seed=3256,StdAdj = 0)
# A tibble: 1 x 8
#   Design GrpSize ObsPHat ObsCliffd ObsStdES PHatType1ER CliffdType1ER StdESType1ER
#   <chr>  <chr>     <dbl>     <dbl>    <dbl>       <dbl>         <dbl>        <dbl>
# 1 2G_g   10        0.498   -0.0034 -0.00464        0.02          0.01         0.02