get_influence_scores {metaggR}

Calculate the Influence Scores

Description

This function computes and plots the influence scores described in Palley & Satopää (2021): Boosting the Wisdom of Crowds Within a Single Judgment Problem: Weighted Averaging Based on Peer Predictions. The current version of the paper is available at https://papers.ssrn.com/sol3/Papers.cfm?abstract_id=3504286

Usage

get_influence_scores(E, P, plotIt = FALSE, cutoff = 7/2)

Arguments

E

Vector of J \ge 6 estimates of the outcome.

P

Vector of J \ge 6 predictions of others. The values must be in the same order as the estimates in E. Specifically, for all j = 1, ..., J, E[j] and P[j] give the jth judge's estimate and prediction of others, respectively.

plotIt

A boolean value. If TRUE, then the function call produces two side-by-side plots:

1. Left plot: This is a scatter plot of the judges' estimates against the judges' implied predictions of others. This plot includes regression lines both with (solid black) and without (dashed red) the exceptionally influential judges. All exceptionally influential judges are shown in red. The knowledge-weighted estimate is shown both with (black square) and without (red circle) exceptionally influential judges.

2. Right plot: This shows the judges' influence scores. All exceptionally influential judges are shown in red. The dashed horizontal line represents the threshold, defined as the user-defined cutoff value x the interquartile range of all influence scores.

For more information on the plots, see the Electronic Companion of Palley & Satopää (2021): Boosting the Wisdom of Crowds Within a Single Judgment Problem: Weighted Averaging Based on Peer Predictions at https://papers.ssrn.com/sol3/Papers.cfm?abstract_id=3504286.

cutoff

A positive scalar describing the cutoff value for the outlier-robust knowledge-weighted estimate. The outlier-robust version calculates the influence scores for all judges. Each influence score is then compared against cutoff x the interquartile range of all influence scores. If a judge's influence score is above this quantity, then that judge is deemed exceptionally influential. This parameter only has an effect if plotIt has been set to TRUE.

Value

J vector of influence scores. Intuitively, the influence score of a judge represents the amount by which the knowledge-weighted estimate would change if that judge was removed from the crowd. Judges with an exceptionally high influence should be inspected. As a default cutoff value, the authors recommend 7/2 times the interquartile range of the individual judges' influence scores.

Examples

# Illustration on the Three Gorges Dam Example in Palley & Satopää (2021):

# The original example with 6 judges is augmented with a 7th judge with an extreme response.
# Judges' estimates:
E2 = c(50, 134, 206, 290, 326, 374, 1000)
# Judges' predictions of others
P2 = c(26, 92, 116, 218, 218, 206, 400)

# The influence score of the 7th judge is much higher than the other judges' scores.
# This judge's response should be inspected and potentially excluded from
# the final knowledge-weighted estimate.
get_influence_scores(E2,P2)