randomCAT {catR}R Documentation

Random generation of adaptive tests (dichotomous and polytomous models)

Description

This command generates a response pattern to an adaptive test, for a given item bank (with either dichotomous or polytomous models), a true ability level, and several lists of CAT parameters (starting items, stopping rule, provisional and final ability estimators).

Usage

randomCAT(trueTheta, itemBank, model = NULL, responses = NULL, min.length = 0, 
 	a.stratified = NULL, genSeed = NULL, cbControl = NULL, nAvailable = NULL, 
 	start = list(fixItems = NULL, seed = NULL, nrItems = 1, theta = 0, D = 1, 
 	randomesque = 1, random.seed = NULL, startSelect = "MFI", cb.control = FALSE, 
 	random.cb = NULL), test = list(method = "BM", priorDist = "norm", 
 	priorPar = c(0, 1), weight = "Huber", tuCo = 1, sem.type = "classic", 
 	sem.exact = FALSE, se.ase = 10, range = c(-4, 4), D = 1, parInt = c(-4, 4, 33), 
 	itemSelect = "MFI", infoType = "observed", randomesque = 1, 
 	random.seed = NULL, AP = 1, proRule = "length", proThr = 20, 
 	constantPatt = NULL), stop = list(rule = "length", thr = 20, alpha = 0.05), 
 	final = list(method = "BM", priorDist = "norm", priorPar = c(0, 1), 
 	weight = "Huber", tuCo = 1, sem.type = "classic", sem.exact = FALSE, 
 	range = c(-4, 4), D = 1, parInt = c(-4, 4, 33), alpha = 0.05), 
 	allTheta = FALSE, save.output = FALSE, output = c("path", "name", "csv"))
## S3 method for class 'cat'
print(x, ...)
## S3 method for class 'cat'
plot(x, ci = FALSE, alpha = 0.05, trueTh = TRUE, classThr = NULL, 
 	save.plot = FALSE, save.options = c("path", "name", "pdf"),...)

Arguments

trueTheta

numeric: the value of the true ability level.

itemBank

numeric: a suitable matrix of item parameters (possibly augmented by group membership for content balancing). See Details.

model

either NULL (default) for dichotomous models, or any suitable acronym for polytomous models. Possible values are "GRM", "MGRM", "PCM", "GPCM", "RSM" and "NRM". See Details.

responses

either NULL (default) or a vector of pre-specified item responses with as many components as the rows of itemBank. See Details.

min.length

integer: the minimum number of items to administer default is zero so no minimum test length). The CAT will not stop before the number of administered items equals min.length, even if the stopping rule is satisfied.

a.stratified

either NULL (default), an integer value or a vector of integer values to set up a-stratified sampling. See Details.

genSeed

either a numeric value to fix the random generation of responses pattern or NULL (default). Ignored if responses is not NULL. See Details.

cbControl

either a list of accurate format to control for content balancing, or NULL. See Details.

nAvailable

either a boolean vector indicating which items (denoted by 1's) are available at the start of the test and which (denoted by 0's) are not, or NULL (default). See Details.

start

a list with the options for starting the adaptive test. See Details.

test

a list with the options for provisional ability estimation and next item selection. See Details.

stop

a list with the options of the stopping rule. See Details.

final

a list with the options for final ability estimation. See Details.

allTheta

logical: should all provisional ability estimates and standard errors be computed and returned (including among the starting items)? Default is FALSE, meaning that provisional ability estimates and standard errors are computed only after the selection of the starting items. Ignored if the $nrItems of the start list is equal to zero.

save.output

logical: should the output be saved in an external text file? (default is FALSE).

output

character: a vector of three components. The first component is either the file path to save the output of "path" (default), the second component is the name of the output file, and the third component is the file type, either "txt" or "csv" (default). See Details.

x

an object of class "cat", typically an output of randomCAT function.

ci

logical: should the confidence intervals be plotted for each provisional ability estimate? (default is TRUE).

alpha

numeric: the significance level for provisional confidence intervals (default is 0.05). Ignored if ci is FALSE.

trueTh

logical: should the true ability level be drawn by a horizontal line? (default is TRUE).

classThr

either a numeric value giving the classification threshold to be displayed, or NULL.

save.plot

logical: should the plot be saved in an external figure? (default is FALSE).

save.options

character: a vector of three components. The first component is either the file path or "path" (default), the second component is the name of the output file or ,"name" (default), and the third component is the file extension, either "pdf" (default) or "jpeg". Ignored if save.plot is FALSE. See Details.

...

other generic arguments to be passed to print and plot functions.

Details

The randomCAT function generates an adaptive test using an item bank specified by arguments itemBank and model, and for a given true ability level specified by argument trueTheta.

Dichotomous IRT models are considered whenever model is set to NULL (default value). In this case, itemBank must be a matrix with one row per item and four columns, with the values of the discrimination, the difficulty, the pseudo-guessing and the inattention parameters (in this order). These are the parameters of the four-parameter logistic (4PL) model (Barton and Lord, 1981).

Polytomous IRT models are specified by their respective acronym: "GRM" for Graded Response Model, "MGRM" for Modified Graded Response Model, "PCM" for Partical Credit Model, "GPCM" for Generalized Partial Credit Model, "RSM" for Rating Scale Model and "NRM" for Nominal Response Model. The itemBank still holds one row per item, end the number of columns and their content depends on the model. See genPolyMatrix for further information and illustrative examples of suitable polytomous item banks.

By default all item responses will be randomly drawn from parent distribution set by the item bank parameters of the itemBank matrix (using the genPattern function for instance). Moreover, the random generation of the item responses can be fixed (for e.g., replication purposes) by assigning some numeric value to the genSeed argument. By default this argument is equal to NULL so the random seed is not fixed (and two successive runs of randomCAT will usually lead to different response patterns).

It is possible, however, to provide a full response pattern of previously recorded responses to each of the item bank, for instance for post-hoc simulations. This is done by providing to the responses argument a vector of binary entries (without missing values). By default responses is set to NULL and item responses will be drawn from the item bank parameters.

With the aforementioned item bank structures, content balancing cannot be controled and cbControl must be set to NULL (default value). Otherwise this will most often lead to an error. In order to allow for content balancing control:

  1. the itemBank must be updated with an additional column holding the group names;

  2. the cbControl argument must be set properly as a list with group names and theoretical proportions for content balancing.

See the nextItem function for further details on how to specify cbControl properly and under which conditions it is operational (see Kingsbury and Zara, 1989, for further details). Separation of the item parameters and the vector of group membership is performed internally through the breakBank function (and thus should not be performed prior to CAT generation).

An alternative method to control for content balancing is to performed a-stratified sampling (Chang and Ying, 1999). This is specified by providing the a.stratified argument either an integer value (i.e., the number of strata) or a vector of integer values (i.e., the number of items per strata). The value to be provided corresponds to the K argument of the aStratified function. Note that this is allowed only with dichotomous IRT models and polytomous GRM, MGRM and GPCM models. By default, a.stratified is NULL and a-stratified sampling is not performed.

The test specification is made by means of four lists of options: one list for the selection of the starting items, one list with the options for provisional ability estimation, one list to define the stopping rule, and one list with the options for final ability estimation. These lists are specified respectively by the arguments start, test, stop and final.

The start list can contain one or several of the following arguments:

These arguments are passed to the function startItems to select the first items of the test.

If the argument nrItems is set to zero, then no starting item is selected and the adaptive process starts with a provisional ability level equal to the value of argument theta (or its default). Moreover, the likelihood function is then set as a flat, uniform function on the whole ability range. See the nextItem function for further details.

The test list can contain one or several of the following arguments:

These arguments are passed to the functions thetaEst and semTheta to estimate the ability level and the standard error of this estimate. In addition, some arguments are passed to nextItem to select the next item appropriately.

The stop list can contain one or several of the following arguments:

Eventually, the final list can contain one or several arguments of the test list (with possiblly different values), as well as the additional alpha argument. The latter specifies the \alpha level of the final confidence interval of ability, which is computed as

[\hat{\theta}-z_{1-\alpha/2} \; se(\hat{\theta}) ; \hat{\theta}+z_{1-\alpha/2} \; se(\hat{\theta})]

where \hat{\theta} and se(\hat{\theta}) are respectively the ability estimate and its standard error. Note that the argument itemSelect of the test list is not used for final estimation of the ability level, and is therefore not allowed into the final list.

If some arguments of these lists are missing, they are automatically set to their default value. The contents of the lists is checked with the testList function, and the adaptive test is generated only if the lists are adequately defined. Otherwise, a message error is printed. Note that the testList function works for both dichotomous and polytomous models.

Usually the ability estimates and related standard errors are computed right after the administration of the starting items (that is, if k starting items are administered, the first (k-1) ability levels and standard errors are missing). This can however be avoided by fixing the argument allTheta to TRUE (by default it is FALSE). In this case, all provisional ability estimates and standard errors are computed and returned, but in the display of th output file, the first (k-1) abilities and standard errors are printed in parentheses (otherwise they are returned as NA values). Note that allTheta is ignored if no starting item was selected (that is, if argument nrItems of the start list is set to zero).

The output of randomCAT, as displayed by the print.cat function, can be stored in a text file provided that save.output is set to TRUE (the default value FALSE does not execute the storage). In this case, the (output argument mus hold three character values: the path to where the output file must be stored, the name of the output file, and the type of output file. If the path is not provided (i.e. left to its default value "path"), it will be saved in the default working directory. The default name is "name", and the default type is "csv". Any other value yields a text file. See the Examples section for an illustration.

The function plot.cat represents the set of provisional and final ability estimates througghout the test. Corresponding confidence intervals (with confidence level defined by the argument alpha) are also drawn if ci=TRUE (which is not the default value), except when stepsize adjustment was made for constant patterns (as the standard error cannot be estimated at this stage). The true ability level can be drawn by a horizontal solid line by specifying trueTh=TRUE (which is the default value); setting it to FALSE will undo the drawing. Finally, any classification threshold can be additionally displayed by specifying a numeric value to the argument classThr. The default value NULL does not display any threshold.

Finally, the plot can be saved in an external file, either as PDF or JPEG format. First, the argument save.plot must be set to TRUE (default is FALSE). Then, the file path for figure storage, the name of the figure and its format are specified through the argument save.options, all as character strings. See the Examples section for further information and a practical example.

Value

The function randomCAT returns a list of class "cat" with the following arguments:

trueTheta

the value of the trueTheta argument.

model

the value of the model argument.

testItems

a vector with the items that were administered during the test.

itemPar

a matrix with the parameters of the items administered during the test.

itemNames

either a vector wit the names of the selected items during the CAT, or NULL.

pattern

the generated response pattern (as vector of 0 and 1 entries).

thetaProv

a vector with the provisional ability estimates.

seProv

a vector with the standard errors of the provisional ability estimates.

ruleFinal

either the stopping rule(s) that was (were) satisfied to make the CAT stop, or NULL.

thFinal

the final ability estimate.

seFinal

the standard error of the final ability estimate.

ciFinal

the confidence interval of the final ability estimate.

min.length

the value of the min.length argument.

a.stratified

the value of the a.stratified argument.

genSeed

the value of the genSeed argument.

startFixItems

the value of the start$fixItems argument (or its default value if missing).

startSeed

the value of the start$seed argument (or its default value if missing).

startNrItems

the value of the start$nrItems argument (or its default value if missing).

startTheta

the value of the start$theta argument (or its default value if missing).

startD

the value of the start$D argument (or its default value if missing).

startRandomesque

the value of the start$randomesque argument (or its default value if missing).

startThStart

the starting ability values used for selecting the first items of the test.

startSelect

the value of the start$startSelect argument (or its default value if missing).

startCB

logical value, being TRUE if both cbControl is not NULL and start$cb.control is TRUE, and FALSE otherwise.

provMethod

the value of the test$method argument (or its default value if missing).

provDist

the value of the test$priorDist argument (or its default value if missing).

provPar

the value of the test$priorPar argument (or its default value if missing).

provWeight

the value of the test$weight argument (or its default value if missing).

provTuCo

the value of the test$tuCo argument (or its default value if missing).

provSemType

the value of the test$sem.type argument (or its default value if missing).

provSemExact

the value of the test$sem.exact argument (or its default value if missing).

se.ase

the value of the test$se.ase argument (or its default value if missing).

provRange

the value of the test$range argument (or its default value if missing).

provD

the value of the test$D argument (or its default value if missing)or NA if model is not NULL.

itemSelect

the value of the test$itemSelect argument (or its default value if missing).

infoType

the value of the test$infoType argument (or its default value if missing). Not returned if model is not NULL.

randomesque

the value of the test$randomesque argument (or its default value if missing).

AP

the value of the test$AP argument (or its default value if missing).

constantPattern

the value of the test$constantPatt argument (or its default value if missing).

cbControl

the value of the cbControl argument (or its default value if missing).

cbGroup

the value of the itemBank$cbGroup element of the item bank itemBank (for dichotomous IRT models), or the cbGroup element returned by the breakBank function (for polytomous IRT models), or NULL.

stopRule

the value of the stop$rule argument (or its default value if missing).

stopThr

the value of the stop$thr argument (or its default value if missing).

stopAlpha

the value of the stop$alpha argument (or its default value if missing).

endWarning

a logical indactor indicating whether the adaptive test stopped because the stopping rule(s) was (were) satisfied, or becasue all items in the bank were administered.

finalMethod

the value of the final$method argument (or its default value if missing).

finalDist

the value of the final$priorDist argument (or its default value if missing).

finalPar

the value of the final$priorPar argument (or its default value if missing).

finalWeight

the value of the final$weight argument (or its default value if missing).

finalTuCo

the value of the final$tuCo argument (or its default value if missing).

finalSemType

the value of the final$sem.type argument (or its default value if missing).

finalSemExact

the value of the final$sem.exact argument (or its default value if missing).

finalRange

the value of the final$range argument (or its default value if missing).

finalD

the value of the final$D argument (or its default value if missing), or NA if model is not NULL.

finalAlpha

the value of the final$alpha argument (or its default value if missing).

save.output

the value of the save.output argument.

output

the value of the output argument.

assigned.responses

a logical value, being TRUE if responses was provided or FALSE responses was set to NULL.

The function print.cat returns similar (but differently organized) results.

Note

When computing the exact standard errors, the number of test items should not be larger than 10 under the 2PL model or any more complex model. This constraint does not hold under the 1PL model.

Author(s)

David Magis
Department of Psychology, University of Liege, Belgium
david.magis@uliege.be

Juan Ramon Barrada
Department of Psychology and Sociology, Universidad Zaragoza, Spain
barrada@unizar.es

References

Barrada, J. R., Olea, J., Ponsoda, V., and Abad, F. J. (2010). A method for the comparison of item selection rules in computerized adaptive testing. Applied Psychological Measurement, 20, 213-229. doi: 10.1177/0146621610370152

Barton, M.A., and Lord, F.M. (1981). An upper asymptote for the three-parameter logistic item-response model. Research Bulletin 81-20. Princeton, NJ: Educational Testing Service.

Birnbaum, A. (1969). Statistical theory for logistic mental test models with a prior distribution of ability. Journal of Mathematical Psychology, 6, 258-276. doi: 10.1016/0022-2496(69)90005-4

Bock, R. D., and Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6, 431-444. doi: 10.1177/014662168200600405

Chang, H.-H., and Ying, Z. (1999). A-stratified multistage computerized adaptive testing. Applied Psychological Measurement, 23, 211-222. doi: 10.1177/01466219922031338

Haley, D.C. (1952). Estimation of the dosage mortality relationship when the dose is subject to error. Technical report no 15. Palo Alto, CA: Applied Mathematics and Statistics Laboratory, Stanford University.

Jeffreys, H. (1939). Theory of probability. Oxford, UK: Oxford University Press.

Jeffreys, H. (1946). An invariant form for the prior probability in estimation problems. Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences, 186, 453-461.

Kingsbury, G. G., and Zara, A. R. (1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education, 2, 359-375. doi: 10.1207/s15324818ame0204_6

Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.

Magis, D. and Barrada, J. R. (2017). Computerized Adaptive Testing with R: Recent Updates of the Package catR. Journal of Statistical Software, Code Snippets, 76(1), 1-18. doi: 10.18637/jss.v076.c01

Magis, D., and Raiche, G. (2012). Random Generation of Response Patterns under Computerized Adaptive Testing with the R Package catR. Journal of Statistical Software, 48 (8), 1-31. doi: 10.18637/jss.v048.i08

Schuester, C., and Yuan, K.-H. (2011). Robust estimation of latent ability in item response models. Journal of Educational and Behavioral Statistics, 36, 720)735.doi: 10.3102/1076998610396890

Thompson, N. A. (2009). Item selection in computerized classification testing. Educational and Psychological Measurement, 69, 778-793. doi: 10.1177/0013164408324460

Urry, V. W. (1970). A Monte Carlo investigation of logistic test models. Unpublished doctoral dissertation. West Lafayette, IN: Purdue University.

van der Linden, W. J. (1998). Bayesian item selection criteria for adaptive testing. Psychometrika, 63, 201-216. doi: 10.1007/BF02294775

Veerkamp, W. J. J., and Berger, M. P. F. (1997). Some new item selection criteria for adaptive testing. Journal of Educational and Behavioral Statistics, 22, 203-226. doi: 10.3102/10769986022002203

Warm, T.A. (1989). Weighted likelihood estimation of ability in item response models. Psychometrika, 54, 427-450. doi: 10.1007/BF02294627

See Also

testList, startItems, nextItem, thetaEst, semTheta, eapEst, eapSem, MWI, MEI, KL, breakBank, genPolyMatrix, genPattern, aStratified

Examples


## Dichotomous models ##

 # Loading the 'tcals' parameters 
 data(tcals)

 # Matrix of item parameters (with no content balancing)
 bank <- as.matrix(tcals[,1:4])
 
 # Creation of a starting list: 5 items with evenly spread theta values
 start <- list(theta = -2:2)

 # Creation of 'test' list: weighted likelihood estimation of provisional ability, 
 # and MFI criterion for next item selection
 test <- list(method = "WL", itemSelect = "MFI")

 # Creation of 'final' list: EAP estimation of final ability
 final <- list(method = "EAP")

 # Creation of a stopping rule: precision criterion, standard error to be reached 0.3
 stop <- list(rule = "precision", thr = 0.3)

 # CAT test
 res <- randomCAT(0, bank, start = start, test = test, stop = stop, final = final)

## Not run: 
 # Update of the stopping rule: by adding a length criterion, with threshold of 10 items
 stop <- list(rule = c("precision", "length"), thr = c(0.3, 10))

 # CAT test
 res <- randomCAT(0, bank, start = start, test = test, stop = stop, final = final)

 # Modifying 'start', 'test' and 'final' lists to compute exact SEs
 start.exact <- list(theta = c(-1, 1))
 test.exact <- list(method = "WL", itemSelect = "MFI", sem.exact = TRUE)
 final.exact <- list(method = "WL", sem.exact = TRUE)
 res.exact <- randomCAT(0, bank, start = start.exact, test = test.exact, 
                        stop = stop, final = final.exact)

 # Creation of a (purely artificial) response pattern for post-hoc simulation
 resp <- rbinom(nrow(bank), 1, 0.5)
 res.ph <- randomCAT(0, bank, start = start, test = test, stop = stop, final = final,
        responses = resp)

 # New 'test' and 'final' rules (BM and EAP estimation with Jeffreys' prior, 
 # randomesque value 5)
 test2 <- list(method = "BM", priorDist = "Jeffreys", randomesque = 5)
 final2 <- list(method = "EAP", priorDist = "Jeffreys")

 # New stopping rule: classification criterion, with classification threshold 0 and 
 # alpha level 0.05
 stop2 <- list(rule = "classification", thr = 0, alpha = 0.05)

 # CAT test with new 'test', 'stop' and 'final' rules
 res2 <- randomCAT(0, bank, start = start, test = test2, stop = stop2, final = final2)

 # New stopping rule: classification criterion, with classification threshold 0.5
 # and alpha level 0.05
 stop3 <- list(rule = "classification", thr = 0.5, alpha = 0.05)

 # CAT test with new 'stop' rule
 res3 <- randomCAT(0, bank, start = start, test = test2, stop = stop3, final=final2)

 # new 'test' and 'stop' rule for next item selection
 test3 <- list(method = "WL", itemSelect = "MLWI")
 stop4 <- list(rule = "length",thr = 10)
 res4 <- randomCAT(0, bank, start = start, test = test3, stop = stop4, final = final2)

 # Creation of the 'cbList' list with arbitrary proportions
 cbList <- list(names = c("Audio1", "Audio2", "Written1", "Written2",
        "Written3"), props = c(0.1, 0.2, 0.2, 0.2, 0.3))

 # CAT test with 'start', 'test', 'stop4' and 'final2' lists and content balancing 
 # using 'cbList' ('tcals' must be used now for content balancing)
 stop4 <- list(rule = "length",thr = 10)
 res5 <- randomCAT(0, tcals, start = start, test = test, stop = stop4, final = final2, 
      cbControl = cbList)

 # new 'start' list to force content balancing control at the starting step
 start2 <- list(seed = 1, nrItems = 3, cb.control = TRUE)
 res6 <- randomCAT(0, tcals, start = start2, test = test, stop = stop4, final = final2, 
      cbControl = cbList)

 # Using progressive item selection rule and requiring all ability estimates and SEs
 test4 <- list(itemSelect = "progressive")
 res6 <- randomCAT(0, tcals, start = start, test = test4, stop = stop4, final = final, 
      cbControl = cbList, allTheta = TRUE)

 # Saving the output in the external 'out' text file within folder 'Program Files' 
 # of hard drive 'C'
 res5 <- randomCAT(0, tcals, start = start, test = test, stop = stop4, final = final2, 
      cbControl = cbList, save.output = TRUE, 
      output = c("c:/Program Files/", "out", "txt"))

 # Plotting results
 plot(res)
 plot(res, ci = TRUE)
 plot(res, ci = TRUE, trueTh = FALSE)
 plot(res, ci = TRUE, classThr = 1)

 # Saving last figure into PDF file 'figure' within folder 'C:/Program Files/'
 plot(res, ci = TRUE, classThr = 1, save.plot = TRUE,
      save.options = c("c:/Program Files/", "figure", "pdf"))

 # With mistake
 plot(res, ci = 0.05)
 plot(res, classThr = TRUE)


## Polytomous models ##

 # Generation of an item bank under GRM with 100 items and at most 4 categories
 m.GRM <- genPolyMatrix(100, 4, "GRM")
 m.GRM <- as.matrix(m.GRM)

 # CAT options
 start <- list(theta = c(-1, 0),startSelect = "MFI")
 test <- list(method = "BM", itemSelect = "KL")
 stop <- list(rule = "precision", thr = 0.4)
 final <- list(method = "EAP")

 # CAT test
 res <- randomCAT(0, m.GRM,model = "GRM", start = start, test = test, stop = stop,
 	final = final)
 res


 # Creation of an appropriate list for content balancing
 # Equal proprotions across subgroups of items
 cbList <- list(names = c("Group1", "Group2", "Group3", "Group4"), props = rep(1,4))

 # With content balancing, all ability estimates and progressive method
 m.GRM <- genPolyMatrix(100, 4, "GRM", cbControl = cbList)
 test <- list(method = "BM", itemSelect = "progressive")
 res <- randomCAT(0, m.GRM, model = "GRM", start = start, test = test, stop = stop, 
 	final = final, cbControl = cbList, allTheta = TRUE)
 res

 # Loading the cat_pav data
 data(cat_pav)
 cat_pav <- as.matrix(cat_pav)

 stop <- list(rule = "length", thr = 10)
 res <- randomCAT(0, cat_pav, model = "GPCM", start = start, test = test, stop = stop, 
 	final = final, allTheta = TRUE)
 res
 
## End(Not run)
[Package catR version 3.17 Index]