cat.evaluation {simCAT}

CAT Evaluation

Description

Evaluate a CAT simulation

Usage

cat.evaluation(results, true.scores, item.name, rmax)

Arguments

results

list with results of a CAT simulation from simCAT

true.scores

true scores

item.name

vector with the name of all items in the bank

rmax

item maximum exposure rate

Value

a list with two elements.

evaluate is a data.frame. Each line corresponds to a replication, and the columns are the following variables:

• rmse root mean square error between true and estimated score

• se standard error of measurement

• correlation correlation between true and estimated score

• bias bias between true and estimated score

• overlap overlap rate

• min_exp minimum exposure rate

• max_exp maximum exposure rate

• n_exp0 number of items not administered

• n_exp_rmax number of items with exposure rate higher than rmax

• length_mean average mean of test length

• length_sd standard deviation of test length

• length_median average median of test length

• min_length minimum test length

• max_length maximum test length

conditional is a data.frame with the same variables (except for length_sd and length_median) conditioned to the true scores. The colnames are the thetas in each decile, that is, quantile(true.scores, probs = seq(.1, 1, length.out = 10)). Each line corresponds to the mean of the investigated variables for each decile. If there are replications, values are the replication means for each decile.

Author(s)

Alexandre Jaloto

Examples

set.seed(1)
n.items <- 50
pars <- data.frame(
 a = rlnorm(n.items),
 b = rnorm(n.items),
 c = rbeta(n.items, 5, 17),
 d = 1)

# thetas
theta <- rnorm(100)

# simulate responses
resps <- gen.resp(theta, pars[,1:3])

results <- simCAT(resps = resps,
 bank = pars[,1:3],
 start.theta = 0,
 sel.method = 'MFI',
 cat.type = 'variable',
 threshold = .3,
 stop = list(se = .3, max.items = 10))

eval <- cat.evaluation(
 results = results,
 true.scores = theta,
 item.name = paste0('I', 1:nrow(pars)),
 rmax = 1)

#### 3 replications
replications <- 3

# simulate responses
set.seed(1)
resps <- list()
for(i in 1:replications)
 resps[[i]] <- gen.resp(theta, pars[,1:3])

# CAT
results <- list()
for (rep in 1:replications)
{
 print(paste0('replication: ', rep, '/', replications))
 results[[rep]] <- simCAT(
  resps = resps[[rep]],
  bank = pars[,1:3],
  start.theta = 0,
  sel.method = 'MFI',
  cat.type = 'variable',
  threshold = .3,
  stop = list(se = .5, max.items = 10))
}

eval <- cat.evaluation(
 results = results,
 true.scores = theta,
 item.name = paste0('I', 1:nrow(pars)),
 rmax = 1)