cdm.est.class.accuracy {CDM}

Classification Reliability in a CDM

Description

This function computes the classification accuracy and consistency originally proposed by Cui, Gierl and Chang (2012; see also Wang et al., 2015). The function computes both statistics by estimators of Johnson and Sinharay (2018; see also Sinharay & Johnson, 2019) and simulation based estimation.

Usage

cdm.est.class.accuracy(cdmobj, n.sims=0, version=2)

Arguments

cdmobj

Object of class din or gdina

n.sims

Number of simulated persons. If n.sims=0, then the number of persons in the original data is used as the sample size. In case of missing item responses, for every simulated dataset this sample size is used.

version

Correct classification reliability statistics can be obtained using the default version=2. For backward compatibility, version=1 contains estimators for CDM (<=6.2) which have been shown to be biased (Johnson & Sinharay, 2018).

Details

The item parameters and the probability distribution of latent classes is used as the basis of the simulation. Accuracy and consistency is estimated for both MLE and MAP classification estimators. In addition, classification accuracy measures are available for the separate classification of all skills.

Value

A data frame for MLE, MAP and MAP (Skill 1, ..., Skill K) classification reliability for the whole latent class pattern and marginal skill classification with following columns:

Pa_est

Classification accuracy (Cui et al., 2012) using the estimator of Johnson and Sinharay, 2018

Pa_sim

Classification accuracy based on simulated data (only for din models)

Pc

Classification consistency (Cui et al., 2012) using the estimator of Johnson and Sinharay, 2018

Pc_sim

Classification consistency based on simulated data (only for din models)

References

Cui, Y., Gierl, M. J., & Chang, H.-H. (2012). Estimating classification consistency and accuracy for cognitive diagnostic assessment. Journal of Educational Measurement, 49, 19-38. doi: 10.1111/j.1745-3984.2011.00158.x

Johnson, M. S., & Sinharay, S. (2018). Measures of agreement to assess attribute-level classification accuracy and consistency for cognitive diagnostic assessments. Journal of Educational Measurement, 45(4), 635-664. doi: 10.1111/jedm.12196

Sinharay, S., & Johnson, M. S. (2019). Measures of agreement: Reliability, classification accuracy, and classification consistency. In M. von Davier & Y.-S. Lee (Eds.). Handbook of diagnostic classification models (pp. 359-377). Cham: Springer. doi: 10.1007/978-3-030-05584-4_17

Wang, W., Song, L., Chen, P., Meng, Y., & Ding, S. (2015). Attribute-level and pattern-level classification consistency and accuracy indices for cognitive diagnostic assessment. Journal of Educational Measurement, 52(4), 457-476. doi: 10.1111/jedm.12096

Examples

## Not run: 
#############################################################################
# EXAMPLE 1: DINO data example
#############################################################################

data(sim.dino, package="CDM")
data(sim.qmatrix, package="CDM")

#***
# Model 1: estimate DINO model with din
mod1 <- CDM::din( sim.dino, q.matrix=sim.qmatrix, rule="DINO")
# estimate classification reliability
cdm.est.class.accuracy( mod1, n.sims=5000)

#***
# Model 2: estimate DINO model with gdina
mod2 <- CDM::gdina( sim.dino, q.matrix=sim.qmatrix, rule="DINO")
# estimate classification reliability
cdm.est.class.accuracy( mod2 )

m1 <- mod1$coef[, c("guess", "slip" ) ]
m2 <- mod2$coef
m2 <- cbind( m1, m2[ seq(1,18,2), "est" ],
          1 - m2[ seq(1,18,2), "est" ]  - m2[ seq(2,18,2), "est" ]  )
colnames(m2) <- c("g.M1", "s.M1", "g.M2", "s.M2" )
  ##   > round( m2, 3 )
  ##          g.M1  s.M1  g.M2  s.M2
  ##   Item1 0.109 0.192 0.109 0.191
  ##   Item2 0.073 0.234 0.072 0.234
  ##   Item3 0.139 0.238 0.146 0.238
  ##   Item4 0.124 0.065 0.124 0.009
  ##   Item5 0.125 0.035 0.125 0.037
  ##   Item6 0.214 0.523 0.214 0.529
  ##   Item7 0.193 0.514 0.192 0.514
  ##   Item8 0.246 0.100 0.246 0.100
  ##   Item9 0.201 0.032 0.195 0.032
# Note that s (the slipping parameter) substantially differs for Item4
# for DINO estimation in 'din' and 'gdina'

## End(Not run)