compRelSEM {semTools}

Composite Reliability using SEM

Description

Calculate composite reliability from estimated factor-model parameters

Usage

compRelSEM(object, obs.var = TRUE, tau.eq = FALSE, ord.scale = TRUE,
  config = character(0), shared = character(0), higher = character(0),
  return.total = FALSE, dropSingle = TRUE, omit.factors = character(0),
  omit.indicators = character(0), omit.imps = c("no.conv", "no.se"),
  return.df = TRUE)

Arguments

object

A lavaan or lavaan.mi object, expected to contain only exogenous common factors (i.e., a CFA model).

obs.var

logical indicating whether to compute AVE using observed variances in the denominator. Setting FALSE triggers using model-implied variances in the denominator.

tau.eq

logical indicating whether to assume (essential) tau-equivalence, yielding a coefficient analogous to \alpha. Setting FALSE yields an \omega-type coefficient.

ord.scale

logical indicating whether to apply Green and Yang's (2009, formula 21) correction, so that reliability is calculated for the actual ordinal response scale (ignored for factors with continuous indicators). Setting FALSE yields coefficients that are only applicable to the continuous latent-response scale.

config

character vector naming any configural constructs in a multilevel CFA. For these constructs (and optional total composite), Lai's (2021) coefficients \omega^\textrm{W} and \omega^\textrm{2L} are returned (or corresponding \alpha coefficients when tau.eq=TRUE), rather than Geldhof et al.'s (2014) coefficients for hypothetical composites of latent components (although the same formula is used for \omega^\textrm{W} in either case). Note that the same name must be used for the factor component represented at each level of the model.

shared

character vector naming any shared constructs in a multilevel CFA. For these constructs (and optional total composite), Lai's (2021) coefficient \omega^\textrm{B} or \alpha^\textrm{B} is returned, rather than Geldhof et al.'s (2014) between-level coefficient for hypothetical composites of latent cluster means. Lai's (2021) coefficient quantifies reliability relative to error associated with both indicators (measurement error) and subjects (sampling error), like a generalizability coefficient. Given that subjects can be considered as raters of their cluster's shared construct, an interrater reliability (IRR) coefficient is also returned, quantifying reliability relative to rater/sampling error alone. To quantify reliability relative to indicator/measurement error alone (i.e., \omega^\textrm{2L}), the shared= construct name(s) can additionally be included in config= argument.

higher

character vector naming any higher-order constructs in object for which composite reliability should be calculated. Ignored when tau.eq=TRUE because alpha is not based on a CFA model; instead, users must fit a CFA with tau-equivalence constraints. To obtain Lai's (2021) multilevel composite-reliability indices for a higher-order factor, do not use this argument; instead, specify the higher-order factor(s) using the shared= or config= argument (compRelSEM will automatically check whether it includes latent indicators and apply the appropriate formula).

return.total

logical indicating whether to return a final column containing the reliability of a composite of all indicators (not listed in omit.indicators) of first-order factors not listed in omit.factors. Ignored in 1-factor models, and should only be set TRUE if all factors represent scale dimensions that could be meaningfully collapsed to a single composite (scale sum or scale mean). Setting a negative value (e.g., -1 returns only the total-composite reliability (excluding coefficients per factor), except when requesting Lai's (2021) coefficients for multilevel configural or shared= constructs.

dropSingle

logical indicating whether to exclude factors defined by a single indicator from the returned results. If TRUE (default), single indicators will still be included in the total column when return.total = TRUE.

omit.factors

character vector naming any common factors modeled in object whose composite reliability is not of interest. For example, higher-order or method factors. Note that reliabilityL2() should be used to calculate composite reliability of a higher-order factor.

omit.indicators

character vector naming any observed variables that should be ignored when calculating composite reliability. This can be useful, for example, to estimate reliability when an indicator is removed.

omit.imps

character vector specifying criteria for omitting imputations from pooled results. Can include any of c("no.conv", "no.se", "no.npd"), the first 2 of which are the default setting, which excludes any imputations that did not converge or for which standard errors could not be computed. The last option ("no.npd") would exclude any imputations which yielded a nonpositive definite covariance matrix for observed or latent variables, which would include any "improper solutions" such as Heywood cases. NPD solutions are not excluded by default because they are likely to occur due to sampling error, especially in small samples. However, gross model misspecification could also cause NPD solutions, users can compare pooled results with and without this setting as a sensitivity analysis to see whether some imputations warrant further investigation.

return.df

logical indicating whether to return reliability coefficients in a data.frame (one row per group/level), which is possible when every model block includes the same factors (after excluding those in omit.factors and applying dropSingle).

Details

Several coefficients for factor-analysis reliability have been termed "omega", which Cho (2021) argues is a misleading misnomer and argues for using \rho to represent them all, differentiated by descriptive subscripts. In our package, we number \omega based on commonly applied calculations.

Bentler (1968) first introduced factor-analysis reliability for a unidimensional factor model with congeneric indicators, labeling the coeficients \alpha. McDonald (1999) later referred to this and other reliability coefficients, first as \theta (in 1970), then as \omega, which is a source of confusion when reporting coefficients (Cho, 2021). Coefficients based on factor models were later generalized to account for multidimenisionality (possibly with cross-loadings) and correlated errors. The general \omega formula implemented in this function is:

\omega = \frac{\left( \sum^{k}_{i = 1} \lambda_i \right)^{2} Var\left( \psi \right)}{\bold{1}^\prime \hat{\Sigma} \bold{1}},

where \hat{\Sigma} can be the model-implied covariance matrix from either the saturated model (i.e., the "observed" covariance matrix, used by default) or from the hypothesized CFA model, controlled by the obs.var argument. A k-dimensional vector \bold{1} is used to sum elements in the matrix. Note that if the model includes any directed effects (latent regression slopes), all coefficients are calculated from total factor variances: lavInspect(object, "cov.lv").

Assuming (essential) tau-equivalence (tau.eq=TRUE) makes \omega equivalent to coefficient \alpha from classical test theory (Cronbach, 1951):

\alpha = \frac{k}{k - 1}\left[ 1 - \frac{\sum^{k}_{i = 1} \sigma_{ii}}{\sum^{k}_{i = 1} \sigma_{ii} + 2\sum_{i < j} \sigma_{ij}} \right],

where k is the number of items in a factor's composite, \sigma_{ii} signifies item i's variance, and \sigma_{ij} signifies the covariance between items i and j. Again, the obs.var argument controls whether \alpha is calculated using the observed or model-implied covariance matrix.

By setting return.total=TRUE, one can estimate reliability for a single composite calculated using all indicators in a multidimensional CFA (Bentler, 1972, 2009). Setting return.total = -1 will return only the total-composite reliability (not per factor).

Higher-Order Factors: The reliability of a composite that represents a higher-order construct requires partitioning the model-implied factor covariance matrix \Phi in order to isolate the common-factor variance associated only with the higher-order factor. Using a second-order factor model, the model-implied covariance matrix of observed indicators \hat{\Sigma} can be partitioned into 3 sources:

1. the second-order common-factor (co)variance: \Lambda \bold{B} \Phi_2 \bold{B}^{\prime} \Lambda^{\prime}

2. the residual variance of the first-order common factors (i.e., not accounted for by the second-order factor): \Lambda \Psi_{u} \Lambda^{\prime}

3. the measurement error of observed indicators: \Theta

where \Lambda contains first-order factor loadings, \bold{B} contains second-order factor loadings, \Phi_2 is the model-implied covariance matrix of the second-order factor(s), and \Psi_{u} is the covariance matrix of first-order factor disturbances. In practice, we can use the full \bold{B} matrix and full model-implied \Phi matrix (i.e., including all latent factors) because the zeros in \bold{B} will cancel out unwanted components of \Phi. Thus, we can calculate the proportion of variance of a composite score calculated from the observed indicators (e.g., a total score or scale mean) that is attributable to the second-order factor (i.e., coefficient \omega):

\omega_{L1}=\frac{\bold{1}^{\prime} \Lambda \bold{B} \Phi \bold{B}^{\prime} \Lambda^{\prime} \bold{1} }{ \bold{1}^{\prime} \hat{\Sigma} \bold{1}},

where \bold{1} is the k-dimensional vector of 1s and k is the number of observed indicators in the composite. Note that a higher-order factor can also have observed indicators.

Categorical Indicators: When all indicators (per composite) are ordinal, the ord.scale argument controls whether the coefficient is calculated on the latent-response scale (FALSE) or on the observed ordinal scale (TRUE, the default). For \omega-type coefficients (tau.eq=FALSE), Green and Yang's (2009, formula 21) approach is used to transform factor-model results back to the ordinal response scale. When ord.scale=TRUE, coefficient \alpha is calculated using the covariance matrix calculated from the integer-valued numeric weights for ordinal categories, consistent with its definition (Chalmers, 2018) and the alpha function in the psych package; this implies obs.var=TRUE, so obs.var=FALSE will be ignored. When ord.scale=FALSE, the standard \alpha formula is applied to the polychoric correlation matrix ("ordinal \alpha"; Zumbo et al., 2007), estimated from the saturated or hypothesized model (see obs.var), and \omega is calculated from CFA results without applying Green and Yang's (2009) correction (see Zumbo & Kroc's, 2019, for a rationalization). No method has been proposed for calculating reliability with a mixture of categorical and continuous indicators, so an error is returned if object includes factors with a mixture of indicator types (unless omitted using omit.factors). If categorical indicators load on a different factor(s) than continuous indicators, then reliability will still be calculated separately for those factors, but return.total must be FALSE (unless omit.factors is used to isolate factors with indicators of the same type).

Multilevel Measurement Models: Under the default settings, compRelSEM() will apply the same formula in each "block" (group and/or level of analysis). In the case of multilevel SEMs, this yields "reliability" for latent within- and between-level components, as proposed by Geldhof et al. (2014). This is not recommended because the coefficients do not correspond to actual composites that would be calculated from the observed data. Lai (2021) proposed coefficients for reliability of actual composites, depending on the type of construct, which requires specifying the names of constructs for which reliability is desired (or multiple constructs whose indicators would compose a multidimensional composite). Configural (config=) and/or shared= constructs can be specified; the same construct can be specified in both arguments, so that overall scale-reliability can be estimated for a shared construct by including it in config. Instead of organizing the output by block (the default), specifying config= and/or shared= will prompt organizing the output by $config and/or $shared.

• The overall (_2L) scale reliability for configural constructs is returned, along with the reliability of a purely individual-level composite (_W, calculated by cluster-mean centering).

• The reliability for a shared construct quantifies generalizability across both indicators and raters (i.e., subjects rating their cluster's construct). Lüdtke et al. (2011) refer to these as measurement error and sampling error, respectively. An interrater reliability (IRR) coefficient is also returned, quantifying generalizability across rater/sampling-error only. To obtain a scale-reliability coefficient (quantifying a shared construct's generalizability across indicator/measurement-error only), include the same factor name in config=. Jak et al. (2021) recommended modeling components of the same construct at both levels, but users may also saturate the within-level model (Lai, 2021).

Be careful about including Level-2 variables in the model, especially whether it makes sense to include them in a total composite for a Level-2 construct. dropSingle=TRUE only prevents estimating reliability for a single-indicator construct, not from including such an indicator in a total composite. It is permissible for shared= constructs to have indicators at Level-2 only. If it is necessary to model other Level-2 variables (e.g., to justify the missing-at-random assumption when using missing = "FIML" estimation), they should be placed in the omit.indicators= argument to exclude them from total composites.

Value

A numeric vector of composite reliability coefficients per factor, or a list of vectors per "block" (group and/or level of analysis), optionally returned as a data.frame when possible (see return.df= argument description for caveat). If there are multiple factors, whose multidimensional indicators combine into a single composite, users can request return.total=TRUE to add a column including a reliability coefficient for the total composite, or return.total = -1 to return only the total-composite reliability (ignored when config= or shared= is specified because each factor's specification must be checked across levels).

Author(s)

Terrence D. Jorgensen (University of Amsterdam; [e-mail])

Uses hidden functions written by Sunthud Pornprasertmanit ([e-mail]) for the old reliability() function.

References

Bentler, P. M. (1968). Alpha-maximized factor analysis (alphamax): Its relation to alpha and canonical factor analysis. Psychometrika, 33(3), 335–345. doi:10.1007/BF02289328

Bentler, P. M. (1972). A lower-bound method for the dimension-free measurement of internal consistency. Social Science Research, 1(4), 343–357. doi:10.1016/0049-089X(72)90082-8

Bentler, P. M. (2009). Alpha, dimension-free, and model-based internal consistency reliability. Psychometrika, 74(1), 137–143. doi:10.1007/s11336-008-9100-1

Chalmers, R. P. (2018). On misconceptions and the limited usefulness of ordinal alpha. Educational and Psychological Measurement, 78(6), 1056–1071. doi:10.1177/0013164417727036

Cho, E. (2021) Neither Cronbach’s alpha nor McDonald’s omega: A commentary on Sijtsma and Pfadt. Psychometrika, 86(4), 877–886. doi:10.1007/s11336-021-09801-1

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. doi:10.1007/BF02310555

Geldhof, G. J., Preacher, K. J., & Zyphur, M. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19(1), 72–91. doi:10.1037/a0032138

Green, S. B., & Yang, Y. (2009). Reliability of summed item scores using structural equation modeling: An alternative to coefficient alpha. Psychometrika, 74(1), 155–167. doi:10.1007/s11336-008-9099-3

Jak, S., Jorgensen, T. D., & Rosseel, Y. (2021). Evaluating cluster-level factor models with lavaan and Mplus. Psych, 3(2), 134–152. doi:10.3390/psych3020012

Lai, M. H. C. (2021). Composite reliability of multilevel data: It’s about observed scores and construct meanings. Psychological Methods, 26(1), 90–102. doi:10.1037/met0000287

Lüdtke, O., Marsh, H. W., Robitzsch, A., & Trautwein, U. (2011). A 2 \times 2 taxonomy of multilevel latent contextual models: Accuracy–bias trade-offs in full and partial error correction models. Psychological Methods, 16(4), 444–467. doi:10.1037/a0024376

McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Erlbaum.

Zumbo, B. D., Gadermann, A. M., & Zeisser, C. (2007). Ordinal versions of coefficients alpha and theta for Likert rating scales. Journal of Modern Applied Statistical Methods, 6(1), 21–29. doi:10.22237/jmasm/1177992180

Zumbo, B. D., & Kroc, E. (2019). A measurement is a choice and Stevens’ scales of measurement do not help make it: A response to Chalmers. Educational and Psychological Measurement, 79(6), 1184–1197. doi:10.1177/0013164419844305

See Also

maximalRelia for the maximal reliability of weighted composite

Examples

data(HolzingerSwineford1939)
HS9 <- HolzingerSwineford1939[ , c("x7","x8","x9")]
HSbinary <- as.data.frame( lapply(HS9, cut, 2, labels=FALSE) )
names(HSbinary) <- c("y7","y8","y9")
HS <- cbind(HolzingerSwineford1939, HSbinary)

HS.model <- ' visual  =~ x1 + x2 + x3
              textual =~ x4 + x5 + x6
              speed   =~ y7 + y8 + y9 '

fit <- cfa(HS.model, data = HS, ordered = c("y7","y8","y9"), std.lv = TRUE)

## works for factors with exclusively continuous OR categorical indicators
compRelSEM(fit)

## reliability for ALL indicators only available when they are
## all continuous or all categorical
compRelSEM(fit, omit.factors = "speed", return.total = TRUE)


## loop over visual indicators to calculate alpha if one indicator is removed
for (i in paste0("x", 1:3)) {
  cat("Drop ", i, ":\n", sep = "")
  print(compRelSEM(fit, omit.factors = c("textual","speed"),
                   omit.indicators = i, tau.eq = TRUE))
}


## Reliability of a composite that represents a higher-order factor
mod.hi <- ' visual  =~ x1 + x2 + x3
            textual =~ x4 + x5 + x6
            speed   =~ x7 + x8 + x9
            general =~ visual + textual + speed '

fit.hi <- cfa(mod.hi, data = HolzingerSwineford1939)
compRelSEM(fit.hi, higher = "general")
## reliabilities for lower-order composites also returned


## works for multigroup models and for multilevel models (and both)
data(Demo.twolevel)
## assign clusters to arbitrary groups
Demo.twolevel$g <- ifelse(Demo.twolevel$cluster %% 2L, "type1", "type2")
model2 <- ' group: type1
  level: 1
    f1 =~ y1 + L2*y2 + L3*y3
    f2 =~ y4 + L5*y5 + L6*y6
  level: 2
    f1 =~ y1 + L2*y2 + L3*y3
    f2 =~ y4 + L5*y5 + L6*y6

group: type2
  level: 1
    f1 =~ y1 + L2*y2 + L3*y3
    f2 =~ y4 + L5*y5 + L6*y6
  level: 2
    f1 =~ y1 + L2*y2 + L3*y3
    f2 =~ y4 + L5*y5 + L6*y6
'
fit2 <- sem(model2, data = Demo.twolevel, cluster = "cluster", group = "g")
compRelSEM(fit2) # Geldhof's indices (hypothetical, for latent components)

## Lai's (2021) indices for Level-1 and configural constructs
compRelSEM(fit2, config = c("f1","f2"))
## Lai's (2021) indices for shared (Level-2) constructs
## (also an interrater reliability coefficient)
compRelSEM(fit2, shared = c("f1","f2"))


## Shared construct using saturated within-level model
mod.sat1 <- ' level: 1
  y1 ~~ y1 + y2 + y3 + y4 + y5 + y6
  y2 ~~ y2 + y3 + y4 + y5 + y6
  y3 ~~ y3 + y4 + y5 + y6
  y4 ~~ y4 + y5 + y6
  y5 ~~ y5 + y6
  y6 ~~ y6

  level: 2
  f1 =~ y1 + L2*y2 + L3*y3
  f2 =~ y4 + L5*y5 + L6*y6
'
fit.sat1 <- sem(mod.sat1, data = Demo.twolevel, cluster = "cluster")
compRelSEM(fit.sat1, shared = c("f1","f2"))


## Simultaneous shared-and-configural model (Stapleton et al, 2016, 2019),
## not recommended, but possible by omitting shared or configural factor.
mod.both <- ' level: 1
    fc =~ y1 + L2*y2 + L3*y3 + L4*y4 + L5*y5 + L6*y6
  level: 2
  ## configural construct
    fc =~ y1 + L2*y2 + L3*y3 + L4*y4 + L5*y5 + L6*y6
  ## orthogonal shared construct
    fs =~ NA*y1 + y2 + y3 + y4 + y5 + y6
    fs ~~ 1*fs + 0*fc
'
fit.both <- sem(mod.both, data = Demo.twolevel, cluster = "cluster")
compRelSEM(fit.both, shared = "fs", config = "fc")