dat.bakdash2021 {metadat}

Dataset on Situation Awareness and Task Performance Associations

Description

Results from 77 papers with 678 effects evaluating associations among measures of situation awareness and task performance.

Usage

dat.bakdash2021

Format

The data frame contains the following columns:

Author	character	paper author(s)
Year	integer	year of paper publication
Title	character	title of paper
DOI	character	digital object identifier (DOI)
DTIC.link	character	permanent link for Defense Technical Information Collection (DITC) reports; see: ⁠https://www.dtic.mil⁠
SA.measure.type	character	type of SA measure
Sample.size	integer	reported sample size
Sample.size.stats	integer	reported sample size based on reported statistics (this reflects excluded participants)
es.z	numeric	z-transformed correlation coefficient; includes ghost results (disclosed and undisclosed non-significant effects not reported in detail) imputed using the draw method described in Bakdash et al. (2021a)
vi.z	numeric	variance for z-transformed correlation (calculated using Sample.size.stats, not Sample.size)
SampleID	character	unique identifier for each experiment/study
Outcome	integer	unique value for each effect size

Details

The dataset contains behavioral experiments from 77 papers/79 studies with a total of 678 effects, evaluating associations among measures of situation awareness (“knowing what is going on”) and task performance. Examples of situation awareness include knowledge of current vehicle speed in a simulated driving task and location and heading of aircraft in a simulated air traffic control task. Corresponding examples of task performance include “the number of collisions in a simulated driving task” and “subject matter expert rating of conflict management in a simulated air control task” (Bakdash et al. 2021a, p. 2). This dataset and the ‘Examples’ are a highly simplified version of the data and code in Bakdash et al. (2021b; 2021c). The journal article by Bakdash et al. (2021a) describes the systematic review and meta-analysis in detail.

This dataset is used to illustrate multilevel multivariate meta-analytic models for the overall pooled effect and pooled effects by situation awareness measure. We also adjust meta-analytic models using cluster-robust variance estimation / cluster-robust inference with the robust function in metafor. Results are shown graphically in a customized forest plot with a prediction interval (estimated plausible range of individual effects). Last, we create a table summarizing the estimated meta-analytic heterogeneity parameters.

The meta-analytic results show most pooled effect sizes in the positive medium range or less. There was also substantial meta-analytic heterogeneity (estimated systematic variance in true effects), nearing the magnitude of the overall pooled effect. We interpret the meta-analytic results as situation awareness typically having limited validity for task performance (i.e., good situation awareness does not tend to have strong probabilistic links with good performance and vice-versa). More formally, measures of situation awareness do not generally and meaningfully capture cognitive processes and other relevant factors underlying task performance.

Run-Time

The code run-time can be greatly sped-up using a linear algebra library with R that makes use of multiple CPU cores. See: https://www.metafor-project.org/doku.php/tips:speeding_up_model_fitting. To measure the run-time, uncomment these three lines: start.time <- Sys.time(), end.time <- Sys.time(), and end.time - start.time. Run-times on Windows 10 x64 with the Intel Math Kernel Library are:

	CPU	Run-Time (Minutes)
	i7-11850H	2.49
	i7-4770	5.38

Concepts

psychology, human factors, engineering, correlation coefficients, multilevel models, multivariate models, cluster-robust inference

Author(s)

Jonathan Bakdash, [e-mail], [e-mail]
Laura Marusich, [e-mail], [e-mail]

Source

Bakdash, J. Z., Marusich, L. R., Cox, K. R., Geuss, M. N., Zaroukian, E. G., & Morris, K. M. (2021b). The validity of situation awareness for performance: A meta-analysis (Code Ocean Capsule). ⁠https://doi.org/10.24433/CO.1682542.v4⁠

Bakdash, J. Z., Marusich, L. R., Cox, K. R., Geuss, M. N., Zaroukian, E. G., & Morris, K. M. (2021c). The validity of situation awareness for performance: A meta-analysis (Systematic Review, Data, and Code). ⁠https://doi.org/10.17605/OSF.IO/4K7ZV⁠

References

Bakdash, J. Z., Marusich, L. R., Cox, K. R., Geuss, M. N., Zaroukian, E. G., & Morris, K. M. (2021a). The validity of situation awareness for performance: A meta-analysis. Theoretical Issues in Ergonomics Science, 1–24. ⁠https://doi.org/10.1080/1463922X.2021.1921310⁠

Supplemental materials: ⁠https://www.tandfonline.com/doi/suppl/10.1080/1463922X.2021.1921310/suppl_file/ttie_a_1921310_sm5524.docx⁠

Examples

### copy data into 'dat' and examine data
dat <- dat.bakdash2021
head(dat[c(1,2,6,8:12)])

## Not run: 
#start.time <- Sys.time()

### load metafor
library(metafor)

### multilevel meta-analytic model to get the overall pooled effect
res.overall <- rma.mv(es.z, vi.z, mods = ~ 1,
                      random = ~ 1 | SampleID / Outcome,
                      data = dat,
                      test = "t")
res.overall

### get prediction interval
predict(res.overall)

### cluster-robust variance estimation (CRVE) / cluster-robust inference
res.overall.crve <- robust(res.overall, cluster = SampleID)
res.overall.crve

### get prediction interval
res.overall.crve.pred <- predict(res.overall.crve)
res.overall.crve.pred

### multilevel meta-analytic model for SA measures
res.sa <-  rma.mv(es.z, vi.z, mods = ~ SA.measure.type - 1,
                  random = ~ 1 | SampleID / Outcome,
                  data = dat,
                  test = "t")
res.sa

### cluster-robust variance estimation (CRVE) / cluster-robust inference
res.sa.crve <- robust(res.sa, cluster = SampleID)
res.sa.crve

### profile likelihood plots
par(mfrow=c(2,1))
profile(res.sa.crve, progbar = FALSE)

### format and combine output of meta-analytic models for the forest plot
all.z        <- c(res.sa.crve$beta,            # SA measures
                  res.overall.crve$beta,       # pooled effect for confidence interval (CI)
                  res.overall.crve$beta)       # pooled effect for prediction interval (PI)

all.ci.lower <- c(res.sa.crve$ci.lb,           # SA measures
                  res.overall.crve.pred$ci.lb, # pooled effect, lower CI
                  res.overall.crve.pred$pi.lb) # pooled effect, lower PI

all.ci.upper <- c(res.sa.crve$ci.ub,           # SA measures
                  res.overall.crve.pred$ci.ub, # pooled effect, upper CI
                  res.overall.crve.pred$pi.ub) # pooled effect, upper PI

### note: there is no p-value for the PI
all.pvals  <- c(res.sa.crve$pval, res.overall.crve$pval)
all.labels <- c(sort(unique(dat$SA.measure.type)), "Overall", "95% Prediction Interval")

### function to round p-values for the forest plot
pvals.round <- function(input) {
  input <- ifelse(input < 0.001, "< 0.001",
           ifelse(input < 0.01, "< 0.01",
           ifelse(input < 0.05 & input >= 0.045, "< 0.05",
           ifelse(round(input, 2) == 1.00, "0.99",
           sprintf("%.2f", round(input, 2))))))}

all.pvals.rounded <- pvals.round(all.pvals)

### forest plot
plot.vals <- data.frame(all.labels, all.z, all.ci.lower, all.ci.upper)

par(mfrow=c(1,1), cex = 1.05)
forest(plot.vals$all.z,
       ci.lb = plot.vals$all.ci.lower,
       ci.ub = plot.vals$all.ci.upper,
       slab  = plot.vals$all.labels,
       psize = 1,
       efac = 0, xlim = c(-1.8, 2.5), clim = c(-1, 1),
       transf = transf.ztor, # transform z to r
       at = seq(-0.5, 1, by = 0.25),
       xlab = expression("Correlation Coefficient"~"("*italic('r')*")"),
       main = "\n\n\nSA Measures",
       ilab = c(all.pvals.rounded, ""), ilab.xpos = 2.45, ilab.pos = 2.5,
       digits = 2, refline = 0, annotate = FALSE)

### keep trailing zero using sprintf
output <- cbind(sprintf("%.2f", round(transf.ztor(plot.vals$all.z), 2)),
                sprintf("%.2f", round(transf.ztor(plot.vals$all.ci.lower), 2)),
                sprintf("%.2f", round(transf.ztor(plot.vals$all.ci.upper), 2)))

### alignment kludge
annotext <- apply(output, 1, function(x) {paste0("  ", x[1], " [", x[2],", ", x[3], "]")})
text( 1.05, 12:1, annotext, pos = 4, cex = 1.05)
text(-1.475, 14.00, "SA Measure", cex = 1.05)
text( 2.30,  14.00, substitute(paste(italic('p-value'))), cex = 1.05)
text( 1.55,  14.00, "Correlation [95% CI]", cex = 1.05)
abline(h = 1.5)

### black polygon for overall mean CIs
addpoly(all.z[11], ci.lb = all.ci.lower[11], ci.ub = all.ci.upper[11],
        rows = 2, annotate = FALSE, efac = 1.5, transf = transf.ztor)

### white polygon for PI
addpoly(all.z[12], ci.lb = all.ci.lower[12], ci.ub = all.ci.upper[12],
        rows = 1, col = "white", border = "black",
        annotate = FALSE, efac = 1.5, transf = transf.ztor)

par(mfrow=c(1,1), cex = 1) # reset graph parameters to default

### confidence intervals for the variance components
re.CI.variances <- confint(res.overall)
re.CI.variances

sigma1.z <- data.frame(re.CI.variances[[1]]["random"])
sigma2.z <- data.frame(re.CI.variances[[2]]["random"])

### fit model using alternative multivariate parameterization
res.overall.alt <- rma.mv(es.z, vi.z, mods = ~ 1,
                          random = ~ factor(Outcome) | factor(SampleID),
                          data = dat,
                          test = "t")

### confidence intervals for the total amount of heterogeneity variance component
res.overall.alt.tau <- confint(res.overall.alt, tau2=1)$random

### I^2: http://www.metafor-project.org/doku.php/tips:i2_multilevel_multivariate
W <- diag(1/dat$vi.z)
X <- model.matrix(res.overall)
P <- W - W %*% X %*% solve(t(X) %*% W %*% X) %*% t(X) %*% W

### I^2 (variance due to heterogeneity): 61%
I2 <- 100 * res.overall.alt$tau2 /
      (res.overall.alt$tau2 + (res.overall$k-res.overall$p)/sum(diag(P)))
I2

### 95% CI for I^2 using uncertainty around tau^2
I2.CI.lb <- 100 * res.overall.alt.tau[1,2] /
            (res.overall.alt.tau[1,2] + (res.overall$k-res.overall$p)/sum(diag(P)))
I2.CI.lb

I2.CI.ub <- 100 * res.overall.alt.tau[1,3] /
            (res.overall.alt.tau[1,3] + (res.overall$k-res.overall$p)/sum(diag(P)))
I2.CI.ub

### total amount of heterogeneity (tau)
sqrt(res.overall.alt$tau2)

### heterogeneity table
table.heterogeneity <- data.frame(matrix(ncol = 3, nrow = 4))
colnames(table.heterogeneity) <- c("Parameter Value",
                                   "Lower 95% CI",
                                   "Upper 95% CI")
rownames(table.heterogeneity) <- c("Tau (Total)",
                                   "Tau1 (Between paper)",
                                   "Tau2 (Within paper)",
                                   "I2 (%)")

table.heterogeneity[1,] <- res.overall.alt.tau[2,]
table.heterogeneity[2,] <- sigma1.z[2,]
table.heterogeneity[3,] <- sigma2.z[2,]
table.heterogeneity[4,] <- c(I2, I2.CI.lb, I2.CI.ub)

round(table.heterogeneity, 2)

#end.time <- Sys.time()
#end.time - start.time


## End(Not run)