detect_tt {aberrance}

Detect test tampering

Description

Detect test tampering at the person level or at the group level.

Usage

detect_tt(
  method,
  psi,
  xi = NULL,
  xi_c = NULL,
  xi_s = NULL,
  x = NULL,
  d = NULL,
  r = NULL,
  x_0 = NULL,
  d_0 = NULL,
  r_0 = NULL,
  interval = c(-4, 4),
  alpha = 0.05,
  group = NULL,
  c = -0.5
)

Arguments

method

The test tampering statistic(s) to compute. Options for score and distractor-based statistics are:

• "EDI_SD_*" for the erasure detection index (Wollack et al., 2015; Wollack & Eckerly, 2017).

• "GBT_SD" for the generalized binomial test statistic (Sinharay & Johnson, 2017). Note: This statistic cannot be computed at the group level.

• "L_SD" for the signed likelihood ratio test statistic (Sinharay et al., 2017). Note: This statistic cannot be computed at the group level.

Options for response-based statistics are:

• "EDI_R_*" for the erasure detection index (Wollack et al., 2015; Wollack & Eckerly, 2017).

• "GBT_R" for the generalized binomial test statistic (Sinharay & Johnson, 2017). Note: This statistic cannot be computed at the group level.

• "L_R" for the signed likelihood ratio test statistic (Sinharay et al., 2017). Note: This statistic cannot be computed at the group level.

Statistics ending in "*" can be computed using various corrections. Options are:

• "*" for all possible corrections.

• "NO" for no correction (Sinharay, 2018; Wollack et al., 2015).

• "CO" for the continuity correction (Wollack et al., 2015; Wollack & Eckerly, 2017). The value of the continuity correction can be specified using c.

• "TS" for the Taylor series expansion (Sinharay, 2018).

psi

A matrix of item parameters.

xi, xi_c, xi_s

Matrices of person parameters. xi is based on all items, xi_c is based on items with changed answers, and xi_s is based on items with the same answers. If NULL (default), person parameters are estimated using maximum likelihood estimation.

x, d, r

Matrices of final data. x is for the item scores, d the item distractors, and r the item responses.

x_0, d_0, r_0

Matrices of initial data. x_0 is for the item scores, d_0 the item distractors, and r_0 the item responses.

interval

The interval to search for the person parameters. Default is c(-4, 4).

alpha

Value(s) between 0 and 1 indicating the significance level(s) used for flagging. Default is 0.05.

group

A vector indicating group membership. If NULL (default), statistics are computed at the person level.

c

Use with the erasure detection index. A value indicating the continuity correction. Default is -0.5.

Value

A list is returned with the following elements:

stat

A matrix of test tampering detection statistics.

pval

A matrix of p-values.

flag

An array of flagging results. The first dimension corresponds to persons/groups, the second dimension to methods, and the third dimension to significance levels.

References

Sinharay, S., Duong, M. Q., & Wood, S. W. (2017). A new statistic for detection of aberrant answer changes. Journal of Educational Measurement, 54(2), 200–217.

Sinharay, S., & Johnson, M. S. (2017). Three new methods for analysis of answer changes. Educational and Psychological Measurement, 77(1), 54–81.

Sinharay, S. (2018). Detecting fraudulent erasures at an aggregate level. Journal of Educational and Behavioral Statistics, 43(3), 286–315.

Wollack, J. A., Cohen, A. S., & Eckerly, C. A. (2015). Detecting test tampering using item response theory. Educational and Psychological Measurement, 75(6), 931–953.

Wollack, J. A., & Eckerly, C. A. (2017). Detecting test tampering at the group level. In G. J. Cizek & J. A. Wollack (Eds.), Handbook of quantitative methods for detecting cheating on tests (pp. 214–231). Routledge.

Examples

# Setup for Examples 1 and 2 ------------------------------------------------

# Settings
set.seed(0)     # seed for reproducibility
N <- 500        # number of persons
n <- 40         # number of items
G <- 20         # number of groups

# Create groups
group <- rep(1:G, each = N / G)

# Randomly select 20% tampered groups with 20% tampered persons
cg <- sample(1:G, size = G * 0.20)
cv <- NULL
for (g in cg) {
  cv <- c(cv, sample(which(group == g), size = N / G * 0.20))
}

# Create vectors of indicators (1 = tampered, 0 = non-tampered)
group_ind <- ifelse(1:G %in% cg, 1, 0)
person_ind <- ifelse(1:N %in% cv, 1, 0)

# Generate person parameters for the nested logit model
xi <- MASS::mvrnorm(
  N,
  mu = c(theta = 0.00, eta = 0.00),
  Sigma = matrix(c(1.00, 0.80, 0.80, 1.00), ncol = 2)
)

# Generate item parameters for the nested logit model
psi <- cbind(
  a = rlnorm(n, meanlog = 0.00, sdlog = 0.25),
  b = rnorm(n, mean = 0.00, sd = 1.00),
  c = runif(n, min = 0.05, max = 0.30),
  lambda1 = rnorm(n, mean = 0.00, sd = 1.00),
  lambda2 = rnorm(n, mean = 0.00, sd = 1.00),
  lambda3 = rnorm(n, mean = 0.00, sd = 1.00),
  zeta1 = rnorm(n, mean = 0.00, sd = 1.00),
  zeta2 = rnorm(n, mean = 0.00, sd = 1.00),
  zeta3 = rnorm(n, mean = 0.00, sd = 1.00)
)

# Simulate uncontaminated data
dat <- sim(psi, xi)
x_0 <- x <- dat$x
d_0 <- d <- dat$d

# Simulate 5% random erasures for non-tampered persons
r_0 <- r <- ifelse(x == 1, 4, d)
for (v in setdiff(1:N, cv)) {
  ci <- sample(1:n, size = n * 0.05)
  for (i in ci) {
    r_0[v, i] <- sample((1:4)[-r[v, i]], size = 1)
  }
  x_0[v, ci] <- ifelse(r_0[v, ci] == 4, 1, 0)
  d_0[v, ci] <- ifelse(r_0[v, ci] == 4, NA, r_0[v, ci])
}
rm(r_0, r)

# Modify contaminated data by tampering with 20% of the scores and
# distractors
for (v in cv) {
  ci <- sample(1:n, size = n * 0.20)
  x[v, ci] <- 1
  d[v, ci] <- NA
}

# Example 1: Person-Level Statistics ----------------------------------------

# Detect test tampering
out <- detect_tt(
  method = c("EDI_SD_*", "GBT_SD", "L_SD"),
  psi = psi,
  x = x,
  d = d,
  x_0 = x_0,
  d_0 = d_0
)

# Example 2: Group-Level Statistics -----------------------------------------

# Detect test tampering
out <- detect_tt(
  method = "EDI_SD_*",
  psi = psi,
  x = x,
  d = d,
  x_0 = x_0,
  d_0 = d_0,
  group = group
)