val.prob.ci.2 {CalibrationCurves}  R Documentation 
Calibration performance
Description
The function val.prob.ci.2
is an adaptation of val.prob
from Frank Harrell's rms package,
https://cran.rproject.org/package=rms. Hence, the description of some of the functions of val.prob.ci.2
come from the the original val.prob
.
The key feature of val.prob.ci.2
is the generation of logistic and flexible calibration curves and related statistics.
When using this code, please cite: Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M.J., Steyerberg,
E.W. (2016). A calibration hierarchy for risk models was defined: from utopia to empirical data. Journal of Clinical Epidemiology,
74, pp. 167176
Usage
val.prob.ci.2(
p,
y,
logit,
group,
weights = rep(1, length(y)),
normwt = FALSE,
pl = TRUE,
smooth = c("loess", "rcs", "none"),
CL.smooth = "fill",
CL.BT = FALSE,
lty.smooth = 1,
col.smooth = "black",
lwd.smooth = 1,
nr.knots = 5,
logistic.cal = FALSE,
lty.log = 1,
col.log = "black",
lwd.log = 1,
xlab = "Predicted probability",
ylab = "Observed proportion",
xlim = c(0.02, 1),
ylim = c(0.15, 1),
m,
g,
cuts,
emax.lim = c(0, 1),
legendloc = c(0.5, 0.27),
statloc = c(0, 0.85),
dostats = TRUE,
cl.level = 0.95,
method.ci = "pepe",
roundstats = 2,
riskdist = "predicted",
cex = 0.75,
cex.leg = 0.75,
connect.group = FALSE,
connect.smooth = TRUE,
g.group = 4,
evaluate = 100,
nmin = 0,
d0lab = "0",
d1lab = "1",
cex.d01 = 0.7,
dist.label = 0.04,
line.bins = 0.05,
dist.label2 = 0.03,
cutoff,
las = 1,
length.seg = 1,
y.intersp = 1,
lty.ideal = 1,
col.ideal = "red",
lwd.ideal = 1,
allowPerfectPredictions = FALSE,
argzLoess = alist(degree = 2),
...
)
Arguments
p 
predicted probability 
y 
vector of binary outcomes 
logit 
predicted log odds of outcome. Specify either 
group 
a grouping variable. If numeric this variable is grouped into

weights 
an optional numeric vector of perobservation weights (usually frequencies),
used only if 
normwt 
set to 
pl 

smooth 

CL.smooth 

CL.BT 

lty.smooth 
the linetype of the flexible calibration curve. Default is 
col.smooth 
the color of the flexible calibration curve. Default is 
lwd.smooth 
the line width of the flexible calibration curve. Default is 
nr.knots 
specifies the number of knots for rcsbased calibration curve. The default as well as the highest allowed value is 5. In case the specified number of knots leads to estimation problems, then the number of knots is automatically reduced to the closest value without estimation problems. 
logistic.cal 

lty.log 
if 
col.log 
if 
lwd.log 
if 
xlab 
xaxis label, default is 
ylab 
yaxis label, default is 
xlim , ylim 
numeric vectors of length 2, giving the x and y coordinates ranges (see 
m 
If grouped proportions are desired, average no. observations per group 
g 
If grouped proportions are desired, number of quantile groups 
cuts 
If grouped proportions are desired, actual cut points for constructing
intervals, e.g. 
emax.lim 
Vector containing lowest and highest predicted probability over which to
compute 
legendloc 
if 
statloc 
the "abc" of model performance (Steyerberg et al., 2011)calibration intercept, calibration slope,
and c statisticwill be added to the plot, using statloc as the upper left corner of a box (default is c(0,.85).
You can specify a list or a vector. Use locator(1) for the mouse, 
dostats 
specifies whether and which performance measures are shown in the figure.

cl.level 
if 
method.ci 
method to calculate the confidence interval of the cstatistic. The argument is passed to 
roundstats 
specifies the number of decimals to which the statistics are rounded when shown in the plot. Default is 2. 
riskdist 
Use 
cex , cex.leg 
controls the font size of the statistics ( 
connect.group 
Defaults to 
connect.smooth 
Defaults to 
g.group 
number of quantile groups to use when 
evaluate 
number of points at which to store the 
nmin 
applies when 
d0lab , d1lab 
controls the labels for events and nonevents (i.e. outcome y) for the histograms.
Defaults are 
cex.d01 
controls the size of the labels for events and nonevents. Default is 0.7. 
dist.label 
controls the horizontal position of the labels for events and nonevents. Default is 0.04. 
line.bins 
controls the horizontal (yaxis) position of the histograms. Default is 0.05. 
dist.label2 
controls the vertical distance between the labels for events and nonevents. Default is 0.03. 
cutoff 
puts an arrow at the specified risk cutoff(s). Default is none. 
las 
controls whether yaxis values are shown horizontally (1) or vertically (0). 
length.seg 
controls the length of the histogram lines. Default is 
y.intersp 
character interspacing for vertical line distances of the legend ( 
lty.ideal 
linetype of the ideal line. Default is 
col.ideal 
controls the color of the ideal line on the plot. Default is 
lwd.ideal 
controls the line width of the ideal line on the plot. Default is 
allowPerfectPredictions 
Logical, indicates whether perfect predictions (i.e. values of either 0 or 1) are allowed. Default is 
argzLoess 
a list with arguments passed to the 
... 
Details
When using the predicted probabilities of an uninformative model (i.e. equal probabilities for all observations), the model has no predictive value. Consequently, where applicable, the value of the performance measure corresponds to the worst possible theoretical value. For the ECI, for example, this equals 1 (Edlinger et al., 2022).
Value
An object of type CalibrationCurve
with the following slots:
call 
the matched call. 
stats 
a vector containing performance measures of calibration. 
cl.level 
the confidence level used. 
Calibration 
contains the calibration intercept and slope, together with their confidence intervals. 
Cindex 
the value of the cstatistic, together with its confidence interval. 
warningMessages 
if any, the warning messages that were printed while running the function. 
CalibrationCurves 
The coordinates for plotting the calibration curves. 
Note
In order to make use (of the functions) of the package auRoc, the user needs to install JAGS. However, since our package only uses the
auc.nonpara.mw
function which does not depend on the use of JAGS, we therefore copied the code and slightly adjusted it when
method="pepe"
.
References
Edlinger, M, van Smeden, M, Alber, HF, Wanitschek, M, Van Calster, B. (2022). Risk prediction models for discrete ordinal outcomes: Calibration and the impact of the proportional odds assumption. Statistics in Medicine, 41( 8), pp. 1334– 1360
Qin, G., & Hotilovac, L. (2008). Comparison of nonparametric confidence intervals for the area under the ROC curve of a continuousscale diagnostic test. Statistical Methods in Medical Research, 17(2), pp. 20721
Steyerberg, E.W., Van Calster, B., Pencina, M.J. (2011). Performance measures for prediction models and markers : evaluation of predictions and classifications. Revista Espanola de Cardiologia, 64(9), pp. 788794
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina M., Steyerberg E.W. (2016). A calibration hierarchy for risk models was defined: from utopia to empirical data. Journal of Clinical Epidemiology, 74, pp. 167176
Van Hoorde, K., Van Huffel, S., Timmerman, D., Bourne, T., Van Calster, B. (2015). A splinebased tool to assess and visualize the calibration of multiclass risk predictions. Journal of Biomedical Informatics, 54, pp. 28393
Examples
# Load package
library(CalibrationCurves)
set.seed(1783)
# Simulate training data
X = replicate(4, rnorm(5e2))
p0true = binomial()$linkinv(cbind(1, X) %*% c(0.1, 0.5, 1.2, 0.75, 0.8))
y = rbinom(5e2, 1, p0true)
Df = data.frame(y, X)
# Fit logistic model
FitLog = lrm(y ~ ., Df)
# Simulate validation data
Xval = replicate(4, rnorm(5e2))
p0true = binomial()$linkinv(cbind(1, Xval) %*% c(0.1, 0.5, 1.2, 0.75, 0.8))
yval = rbinom(5e2, 1, p0true)
Pred = binomial()$linkinv(cbind(1, Xval) %*% coef(FitLog))
# Default calibration plot
val.prob.ci.2(Pred, yval)
# Adding logistic calibration curves and other additional features
val.prob.ci.2(Pred, yval, CL.smooth = TRUE, logistic.cal = TRUE, lty.log = 2,
col.log = "red", lwd.log = 1.5)
val.prob.ci.2(Pred, yval, CL.smooth = TRUE, logistic.cal = TRUE, lty.log = 9,
col.log = "red", lwd.log = 1.5, col.ideal = colors()[10], lwd.ideal = 0.5)