knowledge_weighted_estimate {metaggR}

Knowledge-Weighted Estimate

Description

This function computes the knowledge-weighted estimate from Palley & Satopää (2021): Boosting the Wisdom of Crowds Within a Single Judgment Problem: Weighted Averaging Based on Peer Predictions. The current version of the paper is available at https://papers.ssrn.com/sol3/Papers.cfm?abstract_id=3504286.

Usage

knowledge_weighted_estimate(
  E,
  P,
  cutoff = 7/2,
  remove_inf = FALSE,
  no_inf_check = FALSE
)

Arguments

E

Vector of J estimates of the outcome. If influence scores are calculated (i.e., no_inf_check is FALSE), then the function call requires J \ge 6; else the knowledge-weighted estimated requires at least J \ge 5 judges.

P

Vector of J predictions of others. The values must be in the same order as the estimates in E. Specifically, for all j = 1, ..., J, E[j] and P[j] give the jth judge's estimate and prediction of others, respectively.

cutoff

A positive scalar describing the cutoff value for the outlier-robust knowledge-weighted estimate. The outlier-robust version calculates the influence scores for all judges (see get_influence_scores function). Each influence score is then compared against cutoff x the interquartile range of all influence scores. If a judge's influence score is above this quantity, then that judge is deemed exceptionally influential. By default, the influence scores are checked and a warning is given if an exceptionally influential judge is found. To turn off this feature, set no_inf_check to TRUE.

remove_inf

A boolean value. If TRUE, then all exceptionally influential judges are removed before the knowledge-weighted estimate is calculated. If FALSE, then the knowledge-weighted estimate is calculated based on the responses of all J judges.

no_inf_check

A boolean value. If TRUE, then the influence scores are not calculated at any point. This can be helpful to speed up calculations. However, the authors recommend checking for influential judges each time the knowledge weighted estimate is applied.

Value

A singular value representing the knowledge-weighted estimate

Examples

# Illustration on the Three Gorges Dam Example in Palley & Satopää (2021):

# Judges' estimates:
E1 = c(50, 134, 206, 290, 326, 374)
# Judges' predictions of others
P1 = c(26, 92, 116, 218, 218, 206)

# Knowledge-weighted estimate is 329.305
knowledge_weighted_estimate(E1,P1)

# The original example with 6 judges is augmented with a 7th judge with an extreme response.
# Judges' estimates:
E2 = c(50, 134, 206, 290, 326, 374, 1000)
# Judges' predictions of others
P2 = c(26, 92, 116, 218, 218, 206, 400)

# Knowledge-weighted estimate is 630.0491
# The function call warns of exceptionally influential judges.
knowledge_weighted_estimate(E2,P2)

# Calculate the knowledge-weighted estimate without influence score checking.
knowledge_weighted_estimate(E2,P2, no_inf_check = TRUE)

# Calculate the influence scores of the judges.
# This reveals that the 7th judge is highly influential.
get_influence_scores(E2,P2)

# Calculate the outlier-robust knowledge-weighted estimate.
# This removes all highly influential judges, namely judge 7 in this simple example, 
# and returns the knowledge-weighted estimate of the remaining judges' estimates.
# This estimate aligns with the original 329.305
knowledge_weighted_estimate(E2,P2, remove_inf = TRUE)