powersolve_se {OptHoldoutSize}

Standard error matrix for learning curve parameters (power law)

Description

Find approximate standard error matrix for ⁠(a,b,c)⁠ under power law model for learning curve.

Assumes that

⁠y_i= a x_i^-b + c + e, e~N(0,s^2 y_var_i^2)⁠

Standard error can be computed either asymptotically using Fisher information (method='fisher') or boostrapped (method='bootstrap')

These estimate different quantities: the asymptotic method estimates

Var[MLE(a,b,c)|X,y_var]

and the boostrap method estimates

Var[MLE(a,b,c)].

Usage

powersolve_se(
  x,
  y,
  method = "fisher",
  init = c(20000, 2, 0.1),
  y_var = rep(1, length(y)),
  n_boot = 1000,
  seed = NULL,
  ...
)

Arguments

x

X values (typically training set sizes)

y

Y values (typically observed cost per individual/sample)

method

One of 'fisher' (for asymptotic variance via Fisher Information) or 'bootstrap' (for Bootstrap)

init

Initial values of (a,b,c) to start when computing MLE. Default c(20000,2,0.1)

y_var

Optional parameter giving sampling variance of each y value. Defaults to 1.

n_boot

Number of bootstrap resamples. Only used if method='bootstrap'. Defaults to 1000

seed

Random seed for bootstrap resamples. Defaults to NULL.

...

further parameters passed to optim. We suggest specifying lower and upper bounds; since optim is called on (a*1000^-b,b,c), bounds should be relative to this; for instance, lower=c(0,0,0),upper=c(100,3,1)

Value

Standard error matrix; approximate covariance matrix of MLE(a,b,c)

Examples

A_true=10; B_true=1.5; C_true=0.3; sigma=0.1

set.seed(31525)

X=1+3*rchisq(10000,df=5)
Y=A_true*(X^(-B_true)) + C_true + rnorm(length(X),sd=sigma)

# 'Observations' - 100 samples
obs=sample(length(X),100,rep=FALSE)
Xobs=X[obs]; Yobs=Y[obs]

# True covariance matrix of MLE of a,b,c on these x values
ntest=100
abc_mat_xfix=matrix(0,ntest,3)
abc_mat_xvar=matrix(0,ntest,3)
E1=A_true*(Xobs^(-B_true)) + C_true
for (i in 1:ntest) {
  Y1=E1 + rnorm(length(Xobs),sd=sigma)
  abc_mat_xfix[i,]=powersolve(Xobs,Y1)$par # Estimate (a,b,c) with same X

  X2=1+3*rchisq(length(Xobs),df=5)
  Y2=A_true*(X2^(-B_true)) + C_true + rnorm(length(Xobs),sd=sigma)
  abc_mat_xvar[i,]=powersolve(X2,Y2)$par # Estimate (a,b,c) with variable X
}

Ve1=var(abc_mat_xfix) # empirical variance of MLE(a,b,c)|X
Vf=powersolve_se(Xobs,Yobs,method='fisher') # estimated SE matrix, asymptotic

Ve2=var(abc_mat_xvar) # empirical variance of MLE(a,b,c)
Vb=powersolve_se(Xobs,Yobs,method='bootstrap',n_boot=200) # estimated SE matrix, bootstrap

cat("Empirical variance of MLE(a,b,c)|X\n")
print(Ve1)
cat("\n")
cat("Asymptotic variance of MLE(a,b,c)|X\n")
print(Vf)
cat("\n\n")
cat("Empirical variance of MLE(a,b,c)\n")
print(Ve2)
cat("\n")
cat("Bootstrap-estimated variance of MLE(a,b,c)\n")
print(Vb)
cat("\n\n")