create_cat_design {irt}

Computerized Adaptive Test (CAT) Simulation Design

Description

create_cat_design is a helper function for cat_sim and cat_sim_fast functions. It defines the simulation design.

Ideally, there is a design element for each item. So within this design (which is a list), there are $k$ design elements for each potentially administered item. Each of these sub-design elements are also a list.

Usage

create_cat_design(
  ip = NULL,
  title = NULL,
  true_ip = NULL,
  first_item_rule = "fixed_theta",
  first_item_par = list(theta = 0),
  next_item_rule = "mfi",
  next_item_par = NULL,
  ability_est_rule = "eap",
  ability_est_par = NULL,
  final_ability_est_rule = NULL,
  final_ability_est_par = NULL,
  termination_rule = c("min_item", "min_se", "max_item"),
  termination_par = list(min_item = 10, min_se = 0.33, max_item = 20),
  testlet_rules = NULL,
  exposure_control_rule = NULL,
  exposure_control_par = NULL,
  content_bal_rule = NULL,
  content_bal_par = NULL,
  ability_type = "theta"
)

Arguments

ip

An Itempool-class object containing item parameters, content information, etc.

If ip = NULL this means this is an infinite item pool, where b is on demand, c = 0 and a = 1, D = 1.7.

If true_ip argument is NULL, this item pool will be used to generate item responses.

title

A string value representing the title of this CAT design.

true_ip

An Itempool-class object which holds the true values of item pool parameters that will be used to generate item responses. This is an optional argument. If it is NULL and ip is not missing, then, item responses will be generated using ip.

Default: NULL

first_item_rule

The method how the first item is administered. The main effect of this is to select the first item administered to an examinee. If, for example, first item is desired to be a fixed one or randomly selected from the item pool, then set that rule in next_item_rule.

Default: 'fixed_theta'

Possible values and required parameters:

NULL

If no separate first item selection rule is necessary, the first item will be selected using the next_item_rule and it's parameters next_item_par.

"fixed_theta"

Fixed starting value.

Required parameters for first_item_par argument if this rule is selected:

theta

The value of the initial theta estimate.

"theta_range"

An initial theta estimate within min_theta and max_theta will be randomly selected.

Required parameters for first_item_par argument if this rule is selected:

min_theta

Minimum theta value of the interval.

max_theta

Maximum theta value of the interval.

first_item_par

Parameters for the first item rule.

Default: list(theta = 0)

next_item_rule

A vector of length one or length maximum test length which is designating the next item selection rules.

Default: 'mfi'

Note that, currently, if there are testlets in an item pool and a testlet is selected for administration using one of the methods below, all items within that testlet will be administered regardless of the next item selection rule.

Possible values and required parameters:

random

Randomly select items from the item pool. Exposure control rules and parameters will be ignored for this selection rule.

Required parameters: None.

mfi

Maximum Fisher Information.

Required parameters: None.

mepv

Minimum Expected Posterior Variance.

Required Parameters:

"var_calc_method"

Which method to use to calculate the posterior variance. See Equation (4) of Choi and Swartz (2009), Comparison of CAT Criteria for Polytomous Items.

Available options are:

"eap"

Use the variance from expected a posteriori estimation.

"owen"

Use the variance from Owen's Bayesian estimation. For "Rasch", "1PL", "2PL", "3PL" models this is much faster than "eap" option above.

b_optimal

Select item which has item difficulty that is close to the current ability estimate.

Required parameters: None.

fixed

Administer a fixed set of items from the item pool. This is basically a linear fixed length test where the order of items are predefined. Exposure control rules and parameters will be ignored for this selection rule.

Required Parameters:

item_id

A vector of the item IDs that should be administered.

next_item_par

A list of length one or length maximum test length that sets the parameters of next item selection rules. It can also be NULL, in which case no parameters necessary for that next item selection procedure.

Default: NULL

ability_est_rule

A vector of length one or length maximum test length which is designating the next item selection rules.

Default: "eap"

Possible values and required parameters:

"eap"

Expected-a-posteriori. Required parameters:

prior_dist

Distribution of the prior distribution. Available values:

* norm for normal distribution, * unif for uniform distribution.

The default value is norm.

prior_par

A vector of prior parameters.

* For normal distribution c(0, 1), see ?dnorm * For uniform distribution c(-3, 3), see ?dunif

The default value is c(0, 1).

min_theta

Minimum possible value of theta. It is a lower bound.

The default value is -4.

max_theta

Maximum possible value of theta. It is an upper bound.

The default value is 4.

no_of_quadrature

The number of quadrature, more specifically the number of bins the theta range should be divided. The more bins, the more precise (and slower) the estimates will be.

The default value is 50.

"map"

Maximum-a-posteriori (Bayes Modal). Required parameters:

prior_dist

Distribution of the prior distribution. Currently only available value is:

* norm for normal distribution,

The default value is norm.

prior_par

A vector of prior parameters.

* For normal distribution c(0, 1), see ?dnorm * For uniform distribution c(-3, 3), see ?dunif

The default value is c(0, 1).

min_theta

Minimum possible value of theta. It is a lower bound.

The default value is -4.

max_theta

Maximum possible value of theta. It is an upper bound.

The default value is 4.

tol

The tolerance (precision) level of the estimate.

The default value is 0.00001.

"owen"

Owen's Bayesian Estimation Required parameters:

prior_mean

Prior mean value. The default value is 0.

prior_var

Prior variance value.The default value is 1.

"ml"

Maximum likelihood estimation using Newton-Raphson algorithm. If this method is used, the standard error of ability estimates are calculated using the inverse information value at this theta estimate.

Required parameters:

min_theta

Minimum possible value of theta. It is a lower bound. The default value is -4.

max_theta

Maximum possible value of theta. It is an upper bound. The default value is 4.

criterion

This value determines the accuracy of estimates. Smaller values lead more accuracy but the speed of estimation reduces as the value of criterion decreases. The default value is 0.001.

"eap_ml"

Expected-a-posteriori until an imperfect item response string, then switch to Maximum Likelihood estimation. Required parameters:

prior_dist

Distribution of the prior distribution.

Available values:

norm for normal distribution,

unif for uniform distribution.

prior_par

A vector of prior parameters. For normal distribution c(0, 1), see ?dnorm For uniform distribution c(-3, 3), see ?dunif

min_theta

Minimum possible value of theta. It is a lower bound.

max_theta

Maximum possible value of theta. It is an upper bound.

no_of_quadrature

The number of quadrature, more specifically the number of bins the theta range should be divided. The more bins, the more precise (and slower) the estimates will be.

"map_ml"

Maximum-a-posteriori until an imperfect item response string, then switch to Maximum Likelihood estimation. Required parameters:

prior_dist

Distribution of the prior distribution.

Available values:

norm for normal distribution,

prior_par

A vector of prior parameters. For normal distribution c(0, 1), see ?dnorm

min_theta

Minimum possible value of theta. It is a lower bound.

max_theta

Maximum possible value of theta. It is an upper bound.

tol

The tolerance (precision) level of the estimate.

The default value is 0.00001.

"sum_score"

Simple sum score. Required parameters: NULL

ability_est_par

A list of length one or length maximum test length that sets the parameters of ability estimation rules. It can also be NULL.

* If ability_est_rule = "eap" then the default is list(prior_dist = "norm", prior_par = list(mean = 0, sd = 2), min_theta = -4, max_theta = 4) * If ability_est_rule = "owen" then the default is list(prior_mean = 0, prior_var = 1)

If it is NULL, either no parameters necessary for that ability estimation rule or the defaults of that ability selection rule will be selected.

If it is a list of one, it means that the parameters will be the same throughout the test. The names of the list elements will represent the parameter types.

A list of lists with length of maximum test length designate different parameters for different items in the test progress.

final_ability_est_rule

The ability estimation method that will be used to calculate the final ability estimate. The methods and the parameters are the same as ability_est_rule and ability_est_par. Please see those for details.

Default: NULL

final_ability_est_par

A list of parameters that will be used for the method designated by the final_ability_est_rule.

Default: NULL

termination_rule

This parameter determines how CAT algorithm decides terminate the test.

The order of termination rules is important. The algorithm will check the rules in that order. If for example termination_rule = c('min_se', 'max_item'), first whether the SE smaller than a certain value checked and if it is smaller, then even the maximum number of items haven't been administered, test will terminate.

The "min_item" and "max_item" has a special property where, for "min_item", if the number of items administered smaller than min_item, then test will not terminate regardless of whether other rules satisfied. Similarly, for "max_item", if the number of items is larger than max_item, the test will terminate regardless of whether other conditions satisfied or not. If both "min_item" and "max_item" are in termination rules, then, test will end when both conditions satisfied, i.e. when the number of items administered is equal to or larger than max_item value in termination_par.

The "test length" refers to "Item" objects, i.e. individual items not testlets. For example, if an item pool has 10 testlets each having 2 items and 15 standalone items which are not within a testlet, then the test length can go up to 35 (2 x 10 + 15).

Default: c("min_item", "min_se", "max_item")

"termination_rule" should be a vector that composed of the following termination rules:

"min_item"

The minimum number of items should be satisfied. If the number of administered items are equal to or larger than this number test ends.

"max_item"

The maximum number of items should not be exceeded.

. If this is missing, then the item pool size will be set as maximum length.

"min_se"

If the standard error exceeds min_se value, then the test will terminate.

"sprt"

Sequential Probability Ratio Test (SPRT). SPRT tests two hypotheses:

H_0: Examinee's ability \hat \theta = \theta_0

H_1: Examinee's ability \hat \theta = \theta_1

After the administration of each item, the likelihood (or log-likelihood) of the response string is calculated at \theta_0 and \theta_1. The ratio of this likelihood is then compared to two decision points, A and B.

LR = \frac{L(\theta = theta_1)}{\theta = theta_0}

In order to calculate the lower (A) and upper (B) decision points, one needs to set \alpha and \beta. \alpha represents the rate of false positive classification errors (0 < \alpha < 1), i.e. examinees whose true classification is fail but passed at the end of test. \beta is the rate of false negative classification errors (0 < \beta < 1), i.e. examinees whose true classification is pass but failed at the end of test. A and B can be calculated as:

A = \frac{1 - \beta}{\alpha}

B = \frac{\beta}{1 - \alpha}

If LR > A, examinee passes the test and if LR < B examinee fails the test. If B < LR < A, test continues until the maximum number of items reached (or some other test termination criteria satisfied.)

"sprt" termination rule needs termination_par, where the following parameters should be given in a list:

"theta_0"

The highest theta value that the test developer is willing to fail an examinee.

"theta_1"

The lowest theta value that the test developer is willing to pass an examinee.

"alpha"

The rate of false positive classification errors (0 < alpha < 1), i.e. examinees whose true classification is fail but passed at the end of test.

"beta"

The rate of false negative classification errors (0 < beta < 1), i.e. examinees whose true classification is pass but failed at the end of test.

Example: termination_par = list(sprt = list(theta_0 = -.9, theta_1 = -.1, alpha = 0.05, beta = 0.05))

termination_par

A list of termination rule parameters. This is a named list with length equal to the length of termination_rule argument. The names of the list elements should correspond to the elements of termination_rule argument.

Default: list(min_item = 10, min_se = 0.33, max_item = 20)

testlet_rules

A list containing arguments that specify the rules that will be used within a testlet.

The default value is NULL where the following rules will be applied if there is a testlet: list(next_item_rule = "none", termination_rule = "max_item", termination_par = list(max_item = 999)) where if a testlet is selected all items of this testlet is selected (unless the a testlet has more than 999 items.). Each item is selected with the order it appears in the testlet.

It is assumed that items within testlet are administered together. In other words, an item that does not belong to a selected testlet cannot be administered between two items that belong to the same testlet.

The following list elements are available:

next_item_rule

The way item selection is performed within a testlet. Following options are available:

"none"

Items are selected with the order of observed in the testlet.

"mfi"

Maximum Fisher Information. The most informative unadministered item within the testlet at the current ability estimate is selected.

"termination_rule"

The rule that should be satisfied to stop administering items from a testlet. If there are more than one rule, the termination rules will be applied as the order they appear in the termination_rule vector. For example, if termination_rule = c("max_item", "min_se"), then if max_item criteria is met testlet will be terminated without checking for min_se value.

Following options are available:

"max_item"

An integer representing the maximum number of items administered for each testlet. The test will terminate when maximum number of items is reached or there are no items left in the testlet.

"min_se"

A numeric value representing the standard error of ability estimate value to terminate the test. If the standard error exceeds min_se value, then the testlet will terminate. This testlet termination criteria will only be checked if at least one item from the testlet has already been selected.

"termination_par"

The test termination parameters. See the "termination_par" above in the main function for available options.

exposure_control_rule

A vector of length one or length maximum test length which is designating the next item selection rules. It can be NULL in which case there won't be any exposure control.

Default: NULL, No exposure control will be imposed on item selection.

Possible values and required parameters:

NULL

No exposure control.

"randomesque"

Select one of the most informative num_items items.

num_items

The number of items to select from.

"sympson-hetter"

The algorithm of Sympson-Hetter exposure control is explained in Sympson and Hetter (1985).

This method does not require any additional "exposure_control_par" but each item/testlet should have a "misc" slot like the following misc = list(sympson_hetter_k = .75).

When using 'sympson-hetter' exposure control rule, please ensure that there are sufficient number of items with 'sympson_hetter_k' values 1. Otherwise, examinees might not get a complete test and an error might be raised by the simulation function.

exposure_control_par

A list of length one or maximum test length designating the exposure control for each item. If there are no parameters it will be NULL.

Default: NULL

content_bal_rule

Whether a content balancing is imposed on item selection. Default value is NULL, where no content balancing will be imposed on item selection.

Default: NULL

Possible values and required parameters:

NULL

No content balancing.

max_discrepancy

Given a target content distribution, the content with maximum discrepancy with target discrepancy will be administered.

Required parameters:

target_dist

Target content ratios. For example, suppose there are three content areas: Geometry, Algebra and Arithmetic. If the plan for the test is to include 30 Arithmetic items, then, the target_dist should be: c(Geometry = .3, Arithmetic = .2, Algebra = .5). The names in the vector should correspond to the names of the content areas in the item pool. target_dist should include each content area within the item pool for it to work properly. If the sum of the target_dist is larger than 1, it will be converted to ratios.

content_bal_par

Parameters of content_bal_rule. A list, a list of lists or NULL.

Default: NULL

ability_type

The type of ability the test is measuring. By default it is IRT based single 'theta'.

"theta"

Theta for unidimensional IRT models

"multi_theta"

Theta vector for multidimensional IRT models (Not Implemented Yet).

"cdm"

An attribute vector (Not Implemented Yet).

"raw_score"

Raw score (i.e. total score) of an examinee.

Default: "theta"

Value

A cat_design object that holds the test specifications of a CAT.

Author(s)

Emre Gonulates

References

Sympson, J., & Hetter, R. D. (1985). Controlling item-exposure rates in computerized adaptive testing. 973–977.

See Also

cat_sim

Examples

### Example Designs ###
# Fixed length test IRT test with ability estimation EAP-ML
n_items <- 30
ip <- itempool(data.frame(a = runif(n_items, .5, 1.5), b = rnorm(n_items)))
cd <- create_cat_design(ip = ip, next_item_rule = 'random',
                        termination_rule = 'min_item',
                        termination_par = list('min_item' = n_items))
cd
create_cat_design(ip = ip, next_item_rule = 'random')


n_ip <- 55
ip <- itempool(data.frame(a = runif(n_ip, .5, 1.5), b = rnorm(n_ip)))
# Check the default:
create_cat_design()
create_cat_design(ip = ip)

### Termination Rule ###
create_cat_design(
  termination_rule = c('min_item', 'min_se', 'max_item'),
  termination_par = list(min_item = 10, min_se = .33, max_item = 20))

cd <- create_cat_design(ip = ip, termination_rule = c('min_item', 'min_se'),
                        termination_par = list(min_item = 10, min_se = .33))

### Next Item Rule ###
create_cat_design(ip = ip, next_item_rule = 'random', next_item_par = NULL)
create_cat_design(
  ip = ip, termination_rule = c('min_item', 'max_item'),
  termination_par = list(min_item = 20, max_item = 20),
  next_item_rule = 'fixed',
  next_item_par = list(item_id = ip$item_id[1:20]))

# Linear test where all of the items in the item pool administered in the
# same order as item pool
ip <- generate_ip(n = 15)
create_cat_design(
  ip = ip, termination_rule = c('max_item'),
  termination_par = list(max_item = 15),
  next_item_rule = 'fixed')

# Generate an item pool with two testlets and three standalone items and
# administer first seven items as a linear test.
ip <- c(generate_testlet(n = 2, testlet_id = "t1"), generate_ip(n = 3),
        generate_testlet(n = 5, testlet_id = "t2"))
create_cat_design(
  ip = ip, termination_rule = c('max_item'),
  termination_par = list(max_item = 7),
  next_item_rule = 'fixed')


# A linear test where the item order is predefined.
ip1 <- itempool(data.frame(b = rnorm(5)), item_id = paste0("i",1:5))
cd <- create_cat_design(
  ip = ip1,
  next_item_rule = 'fixed',
  next_item_par = list(item_id = c("i3", "i2", "i4", "i5", "i1")),
  ability_est_rule = "eap",
  termination_rule = 'max_item', termination_par = list(max_item = 5))

### Ability Estimation Rule ###
create_cat_design(
  ability_est_rule = 'eap',
  ability_est_par = list(prior_dist = 'unif',
                         prior_par = list(min = -2, max = 2),
                         min_theta = -4, max_theta = 4,
                         no_of_quadrature = 31))
create_cat_design(
  ability_est_rule = 'ml',
  ability_est_par = list(min_theta = -4, max_theta = 4, criterion = 0.01))

### Exposure Control ###
create_cat_design(exposure_control_rule = 'randomesque',
                exposure_control_par = list(num_items = 1))

# 5-4-3-2-1 exposure control
create_cat_design(
  exposure_control_rule = 'randomesque',
  exposure_control_par = lapply(c(5:1, rep(1, 15)),
                                function(x) list(num_items = x)))

### Content Balancing ###
create_cat_design(
  content_bal_rule = 'max_discrepancy',
  content_bal_par = list(target_dist = c(
    Geometry = .3, `Rational Numbers` = .2, Algebra = .5)))