detect_pm {aberrance}

Detect parametric misfit

Description

Detect parametric misfit using person-fit statistics.

Usage

detect_pm(
  method,
  psi,
  xi = NULL,
  x = NULL,
  d = NULL,
  r = NULL,
  y = NULL,
  interval = c(-4, 4),
  alpha = 0.05
)

Arguments

method

The person-fit statistic(s) to compute. Options for score-based statistics are:

• "ECI2_S_*" for the second standardized extended caution index, also known as \zeta_1 (Tatsuoka, 1984; see also Sinharay, 2018b).

• "ECI4_S_*" for the fourth standardized extended caution index, also known as \zeta_2 (Tatsuoka, 1984; see also Sinharay, 2018b).

• "L_S_*" for the standardized log-likelihood statistic (Drasgow et al., 1985).

Options for distractor-based statistics are:

• "L_D_*" for the standardized log-likelihood statistic (Gorney & Wollack, 2023).

Options for score and distractor-based statistics are:

• "L_SD_*" for the log-likelihood statistic (Gorney & Wollack, 2023).

Options for response-based statistics are:

• "L_R_*" for the standardized log-likelihood statistic (Drasgow et al., 1985).

Options for response time-based statistics are:

• "L_T" for the log-likelihood statistic (Sinharay, 2018a).

Options for score and response time-based statistics are:

• "Q_ST_*" for the log-likelihood statistic that combines "L_S_*" and "L_T" (Gorney, Sinharay, & Liu, 2024).

• "L_ST_*" for the standardized log-likelihood statistic (Gorney, Sinharay, & Liu, 2024). Note: This statistic cannot be computed using the "CF", "CS", "EW", "TSCF", "TSCS", or "TSEW" corrections.

Options for response and response time-based statistics are:

• "Q_RT_*" for the log-likelihood statistic that combines "L_R_*" and "L_T" (Gorney, Sinharay, & Liu, 2024).

• "L_RT_*" for the standardized log-likelihood statistic (Gorney, Sinharay, & Liu, 2024). Note: This statistic cannot be computed using the "CF", "CS", "EW", "TSCF", "TSCS", or "TSEW" corrections.

Statistics ending in "*" can be computed using various corrections. Options are:

• "*" for all possible corrections.

• "NO" for no correction.

• "CF" for the Cornish-Fisher expansion (Molenaar & Hoijtink, 1990).

• "CS" for the chi-squared approximation (Molenaar & Hoijtink, 1990).

• "EW" for the Edgeworth expansion (Bedrick, 1997).

• "TS" for the Taylor series expansion (Snijders, 2001; see also Sinharay, 2016a, 2016b).

• "TSCF" for the Taylor series expansion and Cornish-Fisher expansion (Gorney, Sinharay, & Eckerly, 2024; see also Gorney, 2024).

• "TSCS" for the Taylor series expansion and chi-squared approximation (Gorney Sinharay, & Eckerly, 2024; see also Gorney, 2024).

• "TSEW" for the Taylor series expansion and Edgeworth expansion (Gorney Sinharay, & Eckerly, 2024; see also Gorney, 2024).

psi

A matrix of item parameters.

xi

A matrix of person parameters. If NULL (default), person parameters are estimated using maximum likelihood estimation.

x, d, r, y

Matrices of raw data. x is for the item scores, d the item distractors, r the item responses, and y the item log response times.

interval

The interval to search for the person parameters. Default is c(-4, 4).

alpha

Value(s) between 0 and 1 indicating the significance level(s) used for flagging. Default is 0.05.

Value

A list is returned with the following elements:

stat

A matrix of parametric person-fit statistics.

pval

A matrix of p-values.

flag

An array of flagging results. The first dimension corresponds to persons, the second dimension to methods, and the third dimension to significance levels.

References

Bedrick, E. J. (1997). Approximating the conditional distribution of person fit indexes for checking the Rasch model. Psychometrika, 62(2), 191–199.

Drasgow, F., Levine, M. V., & Williams, E. A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38(1), 67–86.

Gorney, K. (2024). Three new corrections for standardized person-fit statistics for tests with polytomous items [Manuscript submitted for publication].

Gorney, K., Sinharay, S., & Eckerly, C. (2024). Efficient corrections for standardized person-fit statistics [Manuscript submitted for publication].

Gorney, K., Sinharay, S., & Liu, X. (2024). Using item scores and response times in person-fit assessment. British Journal of Mathematical and Statistical Psychology, 77(1), 151–168.

Gorney, K., & Wollack, J. A. (2023). Using item scores and distractors in person-fit assessment. Journal of Educational Measurement, 60(1), 3–27.

Molenaar, I. W., & Hoijtink, H. (1990). The many null distributions of person fit indices. Psychometrika, 55(1), 75–106.

Sinharay, S. (2016a). Asymptotic corrections of standardized extended caution indices. Applied Psychological Measurement, 40(6), 418–433.

Sinharay, S. (2016b). Asymptotically correct standardization of person-fit statistics beyond dichotomous items. Psychometrika, 81(4), 992–1013.

Sinharay, S. (2018a). A new person-fit statistic for the lognormal model for response times. Journal of Educational Measurement, 55(4), 457–476.

Sinharay, S. (2018b). Extension of caution indices to mixed-format tests. British Journal of Mathematical and Statistical Psychology, 71(2), 363–386.

Snijders, T. A. B. (2001). Asymptotic null distribution of person fit statistics with estimated person parameter. Psychometrika, 66(3), 331–342.

Tatsuoka, K. K. (1984). Caution indices based on item response theory. Psychometrika, 49(1), 95–110.

See Also

detect_nm() to detect nonparametric misfit.

Examples

# Setup for Examples 1 and 2 ------------------------------------------------

# Settings
set.seed(0)     # seed for reproducibility
N <- 500        # number of persons
n <- 40         # number of items

# Randomly select 10% examinees with preknowledge and 40% compromised items
cv <- sample(1:N, size = N * 0.10)
ci <- sample(1:n, size = n * 0.40)

# Create vector of indicators (1 = misfitting, 0 = fitting)
ind <- ifelse(1:N %in% cv, 1, 0)

# Example 1: Item Scores and Response Times ---------------------------------

# Generate person parameters for the 3PL model and lognormal model
xi <- MASS::mvrnorm(
  N,
  mu = c(theta = 0.00, tau = 0.00),
  Sigma = matrix(c(1.00, 0.25, 0.25, 0.25), ncol = 2)
)

# Generate item parameters for the 3PL model and lognormal model
psi <- cbind(
  a = rlnorm(n, meanlog = 0.00, sdlog = 0.25),
  b = NA,
  c = runif(n, min = 0.05, max = 0.30),
  alpha = runif(n, min = 1.50, max = 2.50),
  beta = NA
)

# Generate positively correlated difficulty and time intensity parameters
psi[, c("b", "beta")] <- MASS::mvrnorm(
  n,
  mu = c(b = 0.00, beta = 3.50),
  Sigma = matrix(c(1.00, 0.20, 0.20, 0.15), ncol = 2)
)

# Simulate uncontaminated data
dat <- sim(psi, xi)
x <- dat$x
y <- dat$y

# Modify contaminated data by changing the item scores and reducing the log
# response times
x[cv, ci] <- rbinom(length(cv) * length(ci), size = 1, prob = 0.90)
y[cv, ci] <- y[cv, ci] * 0.75

# Detect parametric misfit
out <- detect_pm(
  method = c("L_S_TS", "L_T", "Q_ST_TS", "L_ST_TS"),
  psi = psi,
  x = x,
  y = y
)

# Example 2: Polytomous Item Scores -----------------------------------------

# Generate person parameters for the generalized partial credit model
xi <- cbind(theta = rnorm(N, mean = 0.00, sd = 1.00))

# Generate item parameters for the generalized partial credit model
psi <- cbind(
  a = rlnorm(n, meanlog = 0.00, sdlog = 0.25),
  c0 = 0,
  c1 = rnorm(n, mean = -1.00, sd = 0.50),
  c2 = rnorm(n, mean = 0.00, sd = 0.50),
  c3 = rnorm(n, mean = 1.00, sd = 0.50)
)

# Simulate uncontaminated data
x <- sim(psi, xi)$x

# Modify contaminated data by changing the item scores to the maximum score
x[cv, ci] <- 3

# Detect parametric misfit
out <- detect_pm(
  method = c("ECI2_S_TSCF", "ECI4_S_TSCF", "L_S_TSCF"),
  psi = psi,
  x = x
)