krippendorffs.alpha {krippendorffsalpha}

Apply Krippendorff's Alpha.

Description

Apply Krippendorff's Alpha.

Usage

krippendorffs.alpha(
  data,
  level = c("interval", "nominal", "ordinal", "ratio"),
  method = c("analytical", "customary"),
  confint = TRUE,
  verbose = FALSE,
  control = list()
)

Arguments

data

a matrix of scores. Each row corresponds to a unit, each column to a coder.

level

the level of measurement, one of "nominal", "ordinal", "interval", or "ratio"; or a user-defined distance function.

method

the methodology to apply, either "analytical" or "customary".

confint

logical; if TRUE, a confidence interval is computed. For method = "analytical" the interval is a jackknife interval. For method = "customary" the interval is a bootstrap interval.

verbose

logical; if TRUE, various messages are printed to the console. Note that if confint = TRUE a progress bar (pblapply) is displayed (if possible) during the bootstrap or jackknife computation.

control

a list of control parameters.

bootit

the size of the bootstrap sample. This applies when confint = TRUE and method = "customary". Defaults to 1,000.

nodes

the desired number of nodes in the cluster.

parallel

logical; if TRUE (the default), bootstrapping or jackknife estimation is done in parallel (for confint = TRUE).

type

one of the supported cluster types for makeCluster. Defaults to "SOCK".

Details

This is the package's flagship function. It applies the Krippendorff's Alpha methodology for nominal, ordinal, interval, or ratio levels of measurement, and, if desired, produces confidence intervals. Parallel computing is supported, when applicable.

If the level of measurement is nominal, the discrete metric (nominal.dist) is employed by default. If the level of measurement is interval or ordinal, the squared-difference distance function (interval.dist) is employed by default. (For the ordinal level of measurement, using the squared-difference distance function may be inappropriate, in which case the user should supply his/her own distance function.) If the level of measurement is ratio, a ratio distance function (ratio.dist) is applied. Alternatively, the user may supply his/her own distance function. Said function must handle NA's gracefully; see the above mentioned built-in distance functions for examples.

Argument method is used to choose between the customary Alpha methodology and the analytical methodology developed by Hughes: method = "analytical" or method = "customary". For smaller samples Hughes' methodology should be strongly preferred because that approach reduces bias for point estimation and provides much better performing confidence intervals—jackknife intervals, to be precise. For large samples Krippendorff's customary methodology can safely be used for inference, and speeds computation considerably relative to Hughes' jackknife method.

If argument confint is set to TRUE, a confidence interval is computed. For Hughes' methodology a jackknife interval is produced. For the customary methodology a bootstrap interval is produced. The bootstrap is done by resampling, with replacement, the rows of data and then computing the alpha statistic for the resulting matrix. The elements of argument control are used to control the interval computation.

Value

Function krippendorffs.alpha returns an object of class "krippendorffsalpha", which is a list comprising the following elements.

alpha.hat

the estimate of alpha.

boot.sample

when applicable, the bootstrap sample.

call

the matched call.

coders

the number of coders.

confint

the value of argument confint.

control

the list of control parameters.

data

the matrix of scores, where rows represent units and columns represent coders.

eta.hat

when method = "analytical", log(MSA / MSE).

L

when method = "analytical", the lower 95% confidence limit for alpha.

level

the level of measurement, or a user-dfined distance function.

MSA

when method = "analytical", the estimate of between-unit variation.

MSE

the estimate of within-unit variation.

MST

when method = "customary", the estimate of total variation.

method

the value of argument method.

n_

when method = "analytical", the average number of scores per row of the data matrix.

se

when method = "analytical", the jackknife standard error.

U

when method = "analytical", the upper 95% confidence limit for alpha.

units

the number of units.

verbose

the value of argument verbose.

References

Krippendorff, K. (2013). Computing Krippendorff's alpha-reliability. Technical report, University of Pennsylvania.

Hughes, J. (2022). Toward improved inference for Krippendorff's Alpha agreement coefficient. arXiv.

Examples

# The following data were presented in Krippendorff (2013). This example
# applies Hughes' methodology (the default) to these data. A jackknife
# confidence interval is produced (confint = TRUE). The fit is then
# summarized, and a 99% interval is given.

nominal = matrix(c(1,2,3,3,2,1,4,1,2,NA,NA,NA,
                   1,2,3,3,2,2,4,1,2,5,NA,3,
                   NA,3,3,3,2,3,4,2,2,5,1,NA,
                   1,2,3,3,2,4,4,1,2,5,1,NA), 12, 4)
nominal
fit.nom = krippendorffs.alpha(nominal, level = "nominal", confint = TRUE, verbose = TRUE,
                              control = list(parallel = FALSE))
summary(fit.nom)
confint(fit.nom, level = 0.99)