raterprog {EloChoice}

reliability with progressive rater inclusion

Description

reliability with progressive rater inclusion

Usage

raterprog(winner, loser, raterID, runs=100, ratershuffle=1, progbar=TRUE, kval=100,
startvalue=0, normprob=FALSE)

raterprogplot(xdata)

Arguments

winner

character, vector with the IDs of the winning (preferred) stimuli

loser

character, vector with the IDs of the losing (not preferred) stimuli

raterID

a vector (numeric, character, factor) with rater IDs

runs

numeric, number of randomizations

ratershuffle

numeric, number of times rater order is reshuffled/randomized

progbar

logical, should a progress bar be displayed

kval

numeric, k-value, which determines the maximum number of points a stimulus' rating can change after a single rating event, by default 100

startvalue

numeric, start value around which ratings are centered, by default 0

normprob

logical, by default FALSE, which indicates a logistic approach is taken for calculating winning probabilities (see Elo 1978). Alternatively (TRUE), winning probabilities are calculated from a normal distribution

xdata

results from raterprog

Details

raterprog() calculates reliability, increasing the number of raters to be included in the rating process in a step-wise fashion. In the first (and by default only one) run, the first rater is the one that appears first in the data set, and in subsequent steps raters are added by the order in which they occur. If ratershuffle= is set to values larger than 1, the order in which raters are included is randomized.

raterprogplot() plots the matrix resulting from raterprog(). If ratershuffle= is larger than 1, the average reliability index is plotted alongside quartiles and results from the original rater inclusion sequence.

Note that the function currently only calculates the weighted version of the reliability index.

Value

a numeric matrix. Rows correspond to number of raters in the data set, while columns reflect the number of times the rater order is reshuffled.

Author(s)

Christof Neumann after suggestion by TF

References

Clark AP, Howard KL, Woods AT, Penton-Voak IS, Neumann C (2018). “Why rate when you could compare? Using the 'EloChoice' package to assess pairwise comparisons of perceived physical strength.” PloS one, 13(1), e0190393. doi: 10.1371/journal.pone.0190393.

Examples

data("physical")
# limit to 12 raters
physical <- physical[physical$raterID < 14, ]

x <- raterprog(physical$Winner, physical$Loser, physical$raterID, ratershuffle = 1)
raterprogplot(x)

# with multiple orders in which raters are added
x <- raterprog(physical$Winner, physical$Loser, physical$raterID, ratershuffle = 10)
raterprogplot(x)