ICCrestricted {ShinyItemAnalysis}

Range-restricted reliability with intra-class correlation

Description

Function estimating reliability with intra-class correlation for the complete or for the range-restricted sample.

Usage

ICCrestricted(
  Data,
  case,
  var,
  rank = NULL,
  dir = "top",
  sel = 1,
  nsim = 100,
  ci = 0.95,
  seed = NULL
)

Arguments

Data

matrix or data.frame which includes variables describing ID of ratees (specified in case), ratings (specified in var), and (optionally) rank of ratees (specified in rank).

case

character: name of the variable in Data with ID of the ratee (subject or object being evaluated, such as a respondent, proposal, patient, applicant etc.)

var

character: name of the variable in Data with the ratings/scores.

rank

numeric: vector of ranks of ratees. If not provided, rank of ratee is calculated based on average rating based on var variable.

dir

character: direction of range-restriction, available options are "top" (default) or "bottom". Can be an unambiguous abbreviation (i.e., "t" or "b").

sel

numeric: selected number (given > 1) or percentage (given <= 1) of ratees. Default value is 1 (complete dataset).

nsim

numeric: number of simulations for bootstrap confidence interval. Default value is 100.

ci

numeric: confidence interval. Default value is 0.95.

seed

seed for simulations. Default value is NULL, random seed. See lme4::bootMer() for more detail.

Value

A data.frame with the following columns:

n_sel

number of ratees selected/subsetted.

prop_sel

proportion of ratees selected.

dir

direction of range-restriction. NA if range is effectively not restricted (100% used).

VarID

variance due to ratee, "true variance", between-group variance.

VarResid

residual variance.

VarTotal

total variance.

ICC1

single-rater inter-rater reliability.

ICC1_LCI

lower bound of the confidence interval for ICC1.

ICC1_UCI

upper bound of the confidence interval for ICC1.

ICC3

multiple-rater inter-rater reliability.

ICC3_LCI

lower bound of the confidence interval for ICC3.

ICC3_UCI

upper bound of the confidence interval for ICC3.

Author(s)

Patricia Martinkova
Institute of Computer Science of the Czech Academy of Sciences
[e-mail]

Jan Netik
Institute of Computer Science of the Czech Academy of Sciences
[e-mail]

References

Erosheva, E., Martinkova, P., & Lee, C. (2021a). When zero may not be zero: A cautionary note on the use of inter-rater reliability in evaluating grant peer review. Journal of the Royal Statistical Society - Series A. Accepted.

Erosheva, E., Martinkova, P., & Lee, C. (2021b). Supplementary material for When zero may not be zero: A cautionary note on the use of inter-rater reliability in evaluating grant peer review.

Examples

# ICC for the whole sample
ICCrestricted(Data = AIBS, case = "ID", var = "Score", rank = "ScoreRankAdj")

# ICC for the range-restricted sample considering 80% of top ratees
ICCrestricted(
  Data = AIBS, case = "ID", var = "Score", rank = "ScoreRankAdj",
  sel = 0.8
)