detect_rg {aberrance}

Detect rapid guessing

Description

Detect rapid guessing using item-level response time information.

Usage

detect_rg(
  method,
  t,
  x = NULL,
  outlier = 100,
  chance = 0.25,
  thr = 3,
  nt = 10,
  limits = c(0, Inf),
  min_item = 1
)

Arguments

method

The rapid guessing detection method to apply. Options for visual inspection methods are:

• "VI" for the visual inspection method (Schnipke, 1995). Each plot contains a histogram of the item response times.

• "VITP" for the visual inspection with proportion correct method (Lee & Jia, 2014; Ma et al., 2011). Each plot contains a histogram of the item response times, a dashed red line indicating the proportion correct, and a solid red line indicating the chance rate of success (see chance).

Options for threshold methods are:

• "CT" for the custom threshold method (Wise et al., 2004; Wise & Kong, 2005). The thresholds can be specified using thr.

• "NT" for the normative threshold method (Martinez & Rios, 2023; Wise & Ma, 2012). The percentage(s) of the mean item response time can be specified using nt.

Options for visual inspection and threshold methods are:

• "CUMP" for the cumulative proportion correct method (Guo et al., 2016). Each plot contains a histogram of the item response times, a dashed red line indicating the cumulative proportion correct, and a solid red line indicating the chance rate of success (see chance). Note: No thresholds are returned for items for which the cumulative proportion correct is consistently above or below chance.

t, x

Matrices of raw data. t is for the item response times and x the item scores.

outlier

The percentile(s) above which to delete outliers in t. Length must be equal to 1 or equal to the total number of items. Default is 100, for which no response times are identified as outliers to be deleted.

chance

Use with the visual inspection with proportion correct method and the cumulative proportion correct method. Value(s) indicating the chance rate(s) of success. Length must be equal to 1 or equal to the total number of items. Default is 0.25.

thr

Use with the custom threshold method. Value(s) indicating the response time thresholds. Length must be equal to 1 or equal to the total number of items. Default is 3.

nt

Use with the normative threshold method. Value(s) indicating the percentage(s) of the mean item response time to be used as thresholds. If length is equal to 1, one normative threshold is applied to all items (Wise et al., 2004). Else if length is greater than 1, multiple normative thresholds are applied to all items (Martinez & Rios, 2023). Default is 10, for NT10.

limits

Use with threshold methods. A vector of length 2 indicating the minimum and maximum possible thresholds. Default is c(0, Inf).

min_item

The minimum number of items used to identify unmotivated persons. Default is 1.

Value

A list is returned. If a visual inspection method is used, the list contains the following elements:

plots

A list containing one plot per item.

If a threshold method is used, the list contains the following elements:

thr

A vector or matrix of response time thresholds.

flag

A matrix or array of flagging results.

rte

A vector or matrix of response time effort, equal to 1 minus the proportion of flagged responses per person (Wise & Kong, 2005).

rtf

A vector or matrix of response time fidelity, equal to 1 minus the proportion of flagged responses per item (Wise, 2006).

unmotivated

The proportion of unmotivated persons.

References

Guo, H., Rios, J. A., Haberman, S., Liu, O. L., Wang, J., & Paek, I. (2016). A new procedure for detection of students' rapid guessing responses using response time. Applied Measurement in Education, 29(3), 173–183.

Lee, Y.-H., & Jia, Y. (2014). Using response time to investigate students' test-taking behaviors in a NAEP computer-based study. Large-Scale Assessments in Education, 2, Article 8.

Ma, L., Wise, S. L., Thum, Y. M., & Kingsbury, G. (2011, April). Detecting response time threshold under the computer adaptive testing environment [Paper presentation]. National Council of Measurement in Education, New Orleans, LA, United States.

Martinez, A. J., & Rios, J. A. (2023, April). The impact of rapid guessing on model fit and factor-analytic reliability [Paper presentation]. National Council on Measurement in Education, Chicago, IL, United States.

Schnipke, D. L. (1995, April). Assessing speededness in computer-based tests using item response times [Paper presentation]. National Council on Measurement in Education, San Francisco, CA, United States.

Wise, S. L. (2006). An investigation of the differential effort received by items on a low-stakes computer-based test. Applied Measurement in Education, 19(2), 95–114.

Wise, S. L., Kingsbury, G. G., Thomason, J., & Kong, X. (2004, April). An investigation of motivation filtering in a statewide achievement testing program [Paper presentation]. National Council on Measurement in Education, San Diego, CA, United States.

Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183.

Wise, S. L., & Ma, L. (2012, April). Setting response time thresholds for a CAT item pool: The normative threshold method [Paper presentation]. National Council on Measurement in Education, Vancouver, BC, Canada.

Examples

# Setup for Examples 1 to 3 -------------------------------------------------

# Settings
set.seed(0)     # seed for reproducibility
N <- 5000       # number of persons
n <- 40         # number of items

# Randomly select 20% unmotivated persons
cv <- sample(1:N, size = N * 0.20)

# Create vector of indicators (1 = unmotivated, 0 = motivated)
ind <- ifelse(1:N %in% cv, 1, 0)

# Generate person parameters for the 3PL model and lognormal model
xi <- MASS::mvrnorm(
  N,
  mu = c(theta = 0.00, tau = 0.00),
  Sigma = matrix(c(1.00, 0.25, 0.25, 0.25), ncol = 2)
)

# Generate item parameters for the 3PL model and lognormal model
psi <- cbind(
  a = rlnorm(n, meanlog = 0.00, sdlog = 0.25),
  b = NA,
  c = runif(n, min = 0.05, max = 0.30),
  alpha = runif(n, min = 1.50, max = 2.50),
  beta = NA
)

# Generate positively correlated difficulty and time intensity parameters
psi[, c("b", "beta")] <- MASS::mvrnorm(
  n,
  mu = c(b = 0.00, beta = 3.50),
  Sigma = matrix(c(1.00, 0.20, 0.20, 0.15), ncol = 2)
)

# Simulate item scores and response times
dat <- sim(psi, xi)
x <- dat$x
t <- exp(dat$y)

# Modify contaminated data by guessing on 20% of the items
for (v in cv) {
  ci <- sample(1:n, size = n * 0.20)
  x[v, ci] <- rbinom(length(ci), size = 1, prob = 0.25)
  t[v, ci] <- runif(length(ci), min = 1, max = 10)
}

# Example 1: Visual Inspection Methods --------------------------------------

# Detect rapid guessing using the visual inspection method
out <- detect_rg(
  method = "VI",
  t = t,
  outlier = 90
)

# Detect rapid guessing using the visual inspection with proportion correct
# method
out <- detect_rg(
  method = "VITP",
  t = t,
  x = x,
  outlier = 90
)

# Example 2: Threshold Methods ----------------------------------------------

# Detect rapid guessing using the custom threshold method with a common
# three-second threshold
out <- detect_rg(
  method = "CT",
  t = t,
  thr = 3
)

# Detect rapid guessing using the custom threshold method with 10% of the
# median item response time
out <- detect_rg(
  method = "CT",
  t = t,
  thr = apply(t, 2, function(i) 0.10 * median(i))
)

# Detect rapid guessing using the normative threshold method with 10% of the
# mean item response time
out <- detect_rg(
  method = "NT",
  t = t,
  nt = 10
)

# Detect rapid guessing using the normative threshold method with 5 to 35% of
# the mean item response time
out <- detect_rg(
  method = "NT",
  t = t,
  nt = seq(5, 35, by = 5)
)

# Example 3: Visual Inspection and Threshold Methods ------------------------

# Detect rapid guessing using the cumulative proportion correct method
out <- detect_rg(
  method = "CUMP",
  t = t,
  x = x,
  outlier = 90
)