auc_compare {auctestr} | R Documentation |
Compare AUC values using the FBH method.
Description
Apply the FBH method to compare outcome_col
by compare_col
, averaging
over time_col
(due to non-independence) and then over over_col
by
using Stouffer's Method.
Usage
auc_compare(df, compare_values, filter_value, time_col = "time",
outcome_col = "auc", compare_col = "model_id", over_col = "dataset",
n_col = "n", n_p_col = "n_p", n_n_col = "n_n",
filter_col = "model_variant")
Arguments
df |
DataFrame containing |
compare_values |
names of models to compare (character vector of length 2). These should match exactly the names as they appear in compare_col. |
filter_value |
(optional) keep only observations which contain
|
time_col |
name of column in df representing time of observations (z-scores are averaged over time_col within each model/dataset due to non-independence). These can also be other dependent groupings, such as cross-validation folds. |
outcome_col |
name of column in df representing outcome to compare; this should be Area Under the Receiver Operating Characteristic or A' statistic (this method applies specifically to AUC and not other metrics (i.e., sensitivity, precision, F1).. |
compare_col |
name of column in df representing two conditions to compare
(should contain at least 2 unique values; these two values are specified as
|
over_col |
identifier for independent experiments, iterations, etc. over which z-scores for models are to be compared (using Stouffer's Z). |
n_col |
name of column in df with total number of observations in the sample tested by each row. |
n_p_col |
name of column in df with n_p, number of positive observations. |
n_n_col |
name of column in df with n_n, number of negative observations. |
filter_col |
(optional) name of column in df to filter observations on; keep only
observations which contain |
Value
numeric, overall z-score of comparison using the FBH method.
References
Fogarty, Baker and Hudson, Case Studies in the use of ROC Curve Analysis for Sensor-Based Estimates in Human Computer Interaction, Proceedings of Graphics Interface (2005) pp. 129-136.
Stouffer, S.A.; Suchman, E.A.; DeVinney, L.C.; Star, S.A.; Williams, R.M. Jr. The American Soldier, Vol.1: Adjustment during Army Life (1949).
See Also
Other fbh method: fbh_test
,
se_auc
Examples
## load sample experiment data
data(sample_experiment_data)
## compare VariantA of ModelA and ModelB
auc_compare(sample_experiment_data,
compare_values = c('ModelA', 'ModelB'),
filter_value = c('VariantA'),
time_col = 'time',
outcome_col = 'auc',
compare_col = 'model_id',
over_col = 'dataset',
filter_col = 'model_variant')
## compare VariantC of ModelA and ModelB
auc_compare(sample_experiment_data,
compare_values = c('ModelA', 'ModelB'),
filter_value = c('VariantC'),
time_col = 'time',
outcome_col = 'auc',
compare_col = 'model_id',
over_col = 'dataset',
filter_col = 'model_variant')
## compare ModelC, VariantA and VariantB
auc_compare(sample_experiment_data,
compare_values = c('VariantA', 'VariantB'),
filter_value = c('ModelC'),
time_col = 'time',
outcome_col = 'auc',
compare_col = 'model_variant',
over_col = 'dataset',
filter_col = 'model_id')