atalp {ata}

Automated Test Assembly via Linear Constrained Programming

Description

Ingests item metadata jointly with target test form constraints, and can be parametarized to uses either Boolean (0-1) Integer Linear Programming (ILP) or Mixed Integer Linear Programming (MILP) to construct a test form based on the desired objectives. When MILP is desired the selection of the objective function type should be changed.

Usage

atalp( ipool,
      id,
      constraints,
      refine = FALSE,
      permutate = FALSE,
      sorttimes = 999,
      tieselect = -1,
      type = "const",
      verbose = TRUE,
      aprioriadd = NA,
      posthocadd = NA )

Arguments

ipool

Item by characteristic (property) metadata pool.

id

Name of unique item identifier.

constraints

Complex list object identifying the constraints to be applied in the ATA (see makeconstobj for guided process).

refine

Creates a final test form from permutated solutions, refined to attempt a deviation balance between the observed form and the constraints. Option only effective if permutate is TRUE and type = const in which the constraint weights have meaning; Default is FALSE.

permutate

Requests the test form to be assembled by resorting (sorttimes) the metadata and selecting the most frequently occurring item combination satisfying the constraints. Relevant only for type = const; Default is FALSE.

sorttimes

Number of how often the original input metadata should be resorted. Only functional if permutate is TRUE and type = const; default 999, so that sorttimes + main analysis account for a total of 1,000 selection versions.

tieselect

How should tied items be resolved: -1 (default) - do not manipulate items (which allows for identically functioning items to be included), 1 - select the first item in the list of candidates (sensitive to data sorting); not applicable for situations with all categorical constraints only, 0 - randomly select candidate; not applicable for situations with all categorical constraints only.

type

Type of objective function: const - constraint based only (default), parmin - constraint + minimum non-categorical parameter combination, parmax - constraint + maximum non-categorical parameter combination.

verbose

Should progress be printed to the console? Default TRUE.

aprioriadd

Force item addition (via IDs) to test form before ATA, which affects item selection and constraint attainment success (currently not available).

posthocadd

Force item addition (via IDs) to test form after ATA, which affects final form specifications (currently not available).

Value

A complex list object with test assembly specific estimates:

objective

Constrained objective function value.

items_removed

Removed items from item pool when tieselect is not -1.

excluded

Items from pool excluded.

excluded_set

Item sets excluded. Only included if input constobj includes a set_id.

included

Items from pool included in new test form.

included_set

Item sets from pool included in new test form. Only included if input constobj includes a set_id.

final_ids

Final item ids in the test form.

final_setids

Final set ids in the test form. Only included if input constobj includes a set_id.

Author(s)

Michael Chajewski ([e-mail]), Gulsah Gurkan ([e-mail])

References

Chen, P. (2017). Should we stop developing heuristics and only rely on mixed integer programming solvers in automated test assembly? Applied Psychological Measurement, 41, 227-240.

Diao, Q., & van der Linden, W. J. (2011). Automated test assembly using lp_Solve Version 5.5 in R. Applied Psychological Measurement, 35, 398-409.

Shao, C., Liu, S., Yang, H., & Tsai, T. (2019). Automated test assembly using SAS operations research software in a medical licensing examination. Applied Psychological Measurement, 00, 1-15.

van der Linden, W. J. (2005). A comparison of item-selection methods for adaptive tests with content constraints. Journal of Educational Measurement, 42, 283-302.

Examples

# Specifying constraints
constin <- list(
  nI = 5,                                     # Number of items on the future test
  nC = 4,                                     # Number of constraints to be satisfied
  nameC = c("Content_A","Content_B","p","iSx"), # Name of constraint; must be numeric and must
  # reflect variable name in input
  lowerC = c(2, 3, 3.00, 0.50),               # Lower bound total constraint value on ATA form
  upperC = c(2, 3, 3.50, 0.60),               # Upper bound total constraint value on ATA form
  wC = c(1, 1, 1, 1),                         # Constraint weight used for weighted sum of
                                              # (positive) deviations St
  set_id = NA                                 # Aggregation ID for units / sets 
)

# Running atalp
testLP <- atalp(ipool = metadata_example,
                id = "Item",
                constraints = constin)

# Summary of results
summary(testLP)