KL {catIrt}

Calculate Kullback-Leibler Divergence for IRT Models

Description

KL calculates the IRT implementation of Kullback-Leibler divergence for various IRT models given a vector of ability values, a vector/matrix of item responses, an IRT model, and a value indicating the half-width of an indifference region.

Usage

KL( params, theta, delta = .1 )
## S3 method for class 'brm'
KL( params, theta, delta = .1 )
## S3 method for class 'grm'
KL( params, theta, delta = .1 )

Arguments

params

numeric: a vector or matrix of item parameters. If specified as a matrix, the rows must index the items, and the columns must designate the item parameters. Furthermore, if calculating expected information, the number of rows must match the number of columns of resp. The class of params must match the model: either "brm" or "grm". For the binary response model, params must either be a 3-dimensional vector or a 3-column matrix. See Details for more information.

theta

numeric: a vector of ability values, one for each simulee. When performing a classification CAT, theta should be the boundary points for which to choose the next item.

delta

numeric: a scalar or vector indicating the half-width of the indifference KL will estimate the divergence between \theta - \delta and \theta + \delta using \theta + \delta as the "true model." If delta is a vector, then KL will use recycling to make the length of theta and delta match. See Details for more information.

Details

The function KL returns item divergence and test divergence for the binary response model ("brm") and the graded response model ("grm"). KL-divergence is defined as the following:

KL(\theta_2 || \theta_1) = E_{\theta_2}\log\left[\frac{L(\theta_2)}{L(\theta_1)}\right]

where L(\theta) stands for the likelihood of \theta. Essentially, KL-divergence is the expected log-likelihood gain when using the true model in place of an alternative model.

For the binary response model, KL-divergence for an item simplifies to the following:

KL_j(\theta_2 || \theta_1)_j = p_j(\theta_2)\log\left[\frac{p_j(\theta_2)}{p_j(\theta_1)}\right] + [1 - p_j(\theta_2)]\log\left[\frac{1 - p_j(\theta_2)}{1 - p_j(\theta_1)}\right]

where p_{ij} is the probability of response, as indicated in the help page for simIrt

For the graded response model, KL-divergence for an item simplifies to the following:

KL_j(\theta_2 || \theta_1) = \sum_k{P_{jk}(\theta_2)\log\left[\frac{P_{jk}(\theta_2)}{P_{jk}(\theta_1)}\right]}

where P_{jk}(\theta_2) is the probability of \theta_2 responding in category k as indicated in the help page for simIrt. See Eggen (1999) as applied to classification CAT and van der Linden and Pashley (2010) more generally.

Because of the properties of likelihood functions in item response models, test information is simply the sum of the item informations, or:

KL(\theta_2 || \theta_1) = \sum_jKL_j(\theta_2 || \theta_1)

KL is frequently used to select items in a classification CAT where the hypotheses (e.g. being in one category versus another category are well defined). If "being in the upper category" is \theta_2 and "being in the lower category" is \theta_1, then \theta_2 = B + \delta and \theta_1 = B - \delta where B is the boundary separating the lower category from the upper category. Conversely, if using KL to select items in a precision CAT, then \theta_2 = \hat{\theta}_i + \delta and \theta_1 = \hat{\theta}_i where \hat{\theta}_i is the current, best estimate of \theta. See catIrt for more information.

Value

KL, KL.brm, and KL.grm return a list of the following elements:

item

either: (1) a N \times J matrix of item information for each simulee to each item; (2) a J-length vector of item information for one simulee to each item; or (3) an N-length vector of item information for all simulees to one item, depending on the dimensions of params, theta, annd delta.

test

an N-length vector of test information, one for each simulee. Test information is the sum of item information across items. See Details for more information.

Note

Kullback-Leibler divergence in IRT is not true KL divergence, as the expectation is with respect to a model that is not necessarily true. Furthermore, it is not reciprocal, as KL(\theta_1 || \theta_2) \neq KL(\theta_2 || \theta_1). There have been other KL-based item selection measures proposed, including global information. See Eggen (1999) and itChoose.

Author(s)

Steven W. Nydick [e-mail]

References

Eggen, T. J. H. M. (1999). Item selection in adaptive testing with the sequential probability ratio test. Applied Psychological Measurement, 23, 249 – 261.

Kullback, S., & Leibler, R. A. (1951). On information and sufficiency. The Annals of Mathematical Statistics, 22, 79 – 86.

van dr Linden, W. J. & Pashley, P. J. (2010). Item selection and ability estimation in adaptive testing. In W. J. van der Linden & C. A. W. Glas (Eds.), Elements of Adaptive Testing. New York, NY: Springer.

See Also

catIrt, FI, itChoose, simIrt

Examples

#########################
# Binary Response Model #
#########################

## 1 ##
set.seed(888)
# generating random theta:
theta <- rnorm(20)
# generating an item bank under a 3-parameter binary response model:
b.params <- cbind(a = runif(100, .5, 1.5), b = rnorm(100, 0, 2), c = .2)

# you can indicate class of params or extract it from simIrt object:
class(b.params) <- "brm"

# calculating KL information with delta = .1:
k.info1 <- KL(params = b.params, theta = theta, delt = .1)
# changing delta to .2
k.info2 <- KL(params = b.params, theta = theta, delt = .2)

# notice how the overall information has increased when increasing delt:
k.info1$test; k.info2$test

# also compare with Fisher information:
f.info <- FI(params = b.params, theta = theta, type = "expected")

k.info2$test; f.info$test

# Fisher information is much higher because of how it weighs things.

## 2 ##
# we can maximize information at a boundary - say "0":
k.info3 <- KL(params = b.params, theta = 0, delta = .1)
b.params[which.max(k.info3$item), ]

# notice how the a parameter is high while the b parameter is close to
# 0, so item selection is working.

# does Fisher information choose a different item?
f.info2 <- FI(params = b.params, theta = 0, type = "expected")
b.params[which.max(f.info2$item), ]

# nope - although with more items, who knows?


#########################
# Graded Response Model #
#########################

## 1 ##
set.seed(999)
# generating random theta
theta <- rnorm(20)
# generating an item bank under a graded response model:
g.params <- cbind(runif(100, .5, 1.5), rnorm(100), rnorm(100),
                                       rnorm(100), rnorm(100), rnorm(100))
# simulating responses (so that the parameters are ordered - see simIrt)
g.params <- simIrt(theta = theta, params = g.params, mod = "grm")$params[ , -1]

# we can calculate KL information as before, noting that class(g.params) is "grm"
class(g.params)     # so we don't need to set it ourselves

# and now KL info with delt = .1
k.info4 <- KL(theta = theta, params = g.params)

# KL information is higher when more boundaries
k.info4$test
k.info1$test

# Note: k.info1 would be exactly the same if calculated with the "grm"
#       rather than the "brm"

## 2 ##
# we can also maximize information at boundary "0"
k.info5 <- KL(params = g.params, theta = 0, delta = .1)
g.params[which.max(k.info5$item), ]

# notice how the a parameter is high while the b parameters are pretty spread out.

# does Fisher information choose a different item?
f.info3 <- FI(params = g.params, theta = 0, type = "expected")
g.params[which.max(f.info3$item), ]

# nope - although with more items, who knows?