HB.CA {betafunctions}

An Implementation of the Hanson and Brennan Approach to Estimate Classification Consistency and Accuracy based on Observed Test Scores and Test Reliability.

Description

An implementation of what has been come to be known as the "Hanson and Brennan approach" to classification consistency and accuracy, which by employing a compound beta-binomial distribution assumes that true-scores conform to the four-parameter beta distribution, and errors of measurement to a two-term approximation of the compound binomial distribution. Under these assumptions, the expected classification consistency and accuracy of tests can be estimated from observed outcomes and test reliability.

Usage

HB.CA(
  x = NULL,
  reliability,
  cut,
  testlength,
  true.model = "4P",
  truecut = NULL,
  output = c("accuracy", "consistency"),
  failsafe = TRUE,
  l = 0,
  u = 1,
  modelfit = 10
)

Arguments

x

A vector of observed scores, or a list specifying parameter values. If a list is provided, the list entries must be named after the parameters: l and u for the location-, and alpha and beta for the shape parameters of the Beta true-score distribution, and k for the "Lord's k" parameter (see documentation for the Lords.k function).

reliability

The observed-score squared correlation (i.e., proportion of shared variance) with the true-score.

cut

The cutoff value for classifying observations into above/below categories.

testlength

The total number of test items (or maximum possible score). Must be an integer.

true.model

The probability distribution to be fitted to the moments of the true-score distribution. Options are "4P" (default) and "2P", referring to four- and two-parameter Beta distributions. The "4P" method produces a four-parameter Beta distribution with the same first four moments (mean, variance, skewness, and kurtosis) as the estimated true-score distribution, while the "2P" method produces a two-parameter Beta distribution with the first two moments (mean and variance) as the estimated true-score distribution.

truecut

Optional specification of a "true" cutoff. Useful for producing ROC curves (see documentation for the HB.ROC() function).

output

Character vector indicating which types of statistics (i.e, accuracy and/or consistency) are to be computed and included in the output. Permissible values are "accuracy" and "consistency".

failsafe

Logical value indicating whether to engage the automatic fail-safe defaulting to the two-parameter Beta true-score distribution if the four-parameter fitting procedure produces impermissible parameter estimates. Default is TRUE (i.e., the function will engage failsafe if the four-parameter Beta-distribution fitting-procedure produced impermissible estimates).

l

If true.model = "2P" or failsafe = TRUE, the lower-bound location parameter to be used in the two-parameter fitting procedure. Default is 0 (i.e., the lower-bound of the Standard Beta distribution).

u

If true.model = "2P" or failsafe = TRUE, the upper-bound location parameter to be used in the two-parameter fitting procedure. Default is 1 (i.e., the upper-bound of the Standard Beta distribution).

modelfit

Allows for controlling the chi-square test for model fit by setting the minimum bin-size for expected observations. Can alternatively be set to NULL to forego model-fit testing (speeding up the function). In accordance with standard recommendations for chi-square tests the default input to this argument is 10.

Value

A list containing the estimated parameters necessary for the approach (i.e., the effective test-length and the beta distribution parameters), a chi-square test of model-fit, the confusion matrix containing estimated proportions of true/false pass/fail categorizations for a test, diagnostic performance statistics, and / or a classification consistency matrix and indices. Accuracy output includes a confusion matrix and diagnostic performance indices, and consistency output includes a consistency matrix and consistency indices p (expected proportion of agreement between two independent test administrations), p_c (proportion of agreement on two independent administrations expected by chance alone), and Kappa (Cohen's Kappa).

Note

This implementation of the Hanson-Brennan approach is much slower than the implementation of the Livingston and Lewis approach, as there is no native implementation of Lord's two-term approximation to the Compound-Binomial distribution in R. This implementation uses a "brute-force" method of computing the cumulative probabilities from the compound-Binomial distribution, which will by necessity be more resource intensive.

References

Hanson, Bradley A. (1991). Method of Moments Estimates for the Four-Parameter Beta Compound Binomial Model and the Calculation of Classification Consistency Indexes. American College Testing.

Lord. Frederic M. (1965). A Strong True-Score Theory, With Applications. Psychometrika, 30(3).

Lewis, Don and Burke, C. J. (1949). The Use and Misuse of the Chi-Square Test. Psychological Bulletin, 46(6).

Examples

# Generate some fictional data. Say, 1000 individuals take a test with a
# maximum score of 50.
# Generate some fictional data. Say, 1000 individuals take a 20-item test.
set.seed(1234)
p.success <- rBeta.4P(1000, 0.15, 0.85, 6, 4)
 for (i in 1:20) {
   if (i == 1) {
     rawdata <- matrix(nrow = 1000, ncol = 20)
     }
   rawdata[, i] <- rbinom(1000, 1, p.success)
 }

# Suppose the cutoff value for attaining a pass is 10 items correct, and
# that the reliability of this test was estimated using the Cronbach's Alpha
# estimator. To estimate and retrieve the estimated parameters, confusion and
# consistency matrices, and accuracy and consistency indices using HB.CA():
HB.CA(x = rowSums(rawdata), reliability = cba(rawdata), cut = 10,
testlength = 20)