pstRT_rlddm1 {hBayesDM}

Reinforcement Learning Drift Diffusion Model 1

Description

Hierarchical Bayesian Modeling of the Probabilistic Selection Task (with RT data) using Reinforcement Learning Drift Diffusion Model 1. It has the following parameters: a (boundary separation), tau (non-decision time), v (drift rate scaling), alpha (learning rate).

• Task: Probabilistic Selection Task (with RT data) (Frank et al., 2007; Frank et al., 2004)

• Model: Reinforcement Learning Drift Diffusion Model 1 (Pedersen et al., 2017)

Usage

pstRT_rlddm1(
  data = NULL,
  niter = 4000,
  nwarmup = 1000,
  nchain = 4,
  ncore = 1,
  nthin = 1,
  inits = "vb",
  indPars = "mean",
  modelRegressor = FALSE,
  vb = FALSE,
  inc_postpred = FALSE,
  adapt_delta = 0.95,
  stepsize = 1,
  max_treedepth = 10,
  ...
)

Arguments

data

Data to be modeled. It should be given as a data.frame object, a filepath for a tab-seperated txt file, "example" to use example data, or "choose" to choose data with an interactive window. Columns in the dataset must include: "subjID", "cond", "prob", "choice", "RT", "feedback". See Details below for more information.

niter

Number of iterations, including warm-up. Defaults to 4000.

nwarmup

Number of iterations used for warm-up only. Defaults to 1000.

nchain

Number of Markov chains to run. Defaults to 4.

ncore

Number of CPUs to be used for running. Defaults to 1.

nthin

Every i == nthin sample will be used to generate the posterior distribution. Defaults to 1. A higher number can be used when auto-correlation within the MCMC sampling is high.

inits

Character value specifying how the initial values should be generated. Possible options are "vb" (default), "fixed", "random", or your own initial values.

indPars

Character value specifying how to summarize individual parameters. Current options are: "mean", "median", or "mode".

modelRegressor

Whether to export model-based regressors (TRUE or FALSE). For this model they are: "Q1", "Q2".

vb

Use variational inference to approximately draw from a posterior distribution. Defaults to FALSE.

inc_postpred

Include trial-level posterior predictive simulations in model output (may greatly increase file size). Defaults to FALSE. If set to TRUE, it includes: "choice_os", "RT_os", "choice_sm", "RT_sm", "fd_sm"

adapt_delta

Floating point value representing the target acceptance probability of a new sample in the MCMC chain. Must be between 0 and 1. See Details below.

stepsize

Integer value specifying the size of each leapfrog step that the MCMC sampler can take on each new iteration. See Details below.

max_treedepth

Integer value specifying how many leapfrog steps the MCMC sampler can take on each new iteration. See Details below.

...

For this model, it's possible to set model-specific argument(s) as follows:

RTbound

Floating point value representing the lower bound (i.e., minimum allowed) reaction time. Defaults to 0.1 (100 milliseconds).

initQ

Floating point value representing the model's initial value of any choice.

Details

This section describes some of the function arguments in greater detail.

data should be assigned a character value specifying the full path and name (including extension information, e.g. ".txt") of the file that contains the behavioral data-set of all subjects of interest for the current analysis. The file should be a tab-delimited text file, whose rows represent trial-by-trial observations and columns represent variables.
For the Probabilistic Selection Task (with RT data), there should be 6 columns of data with the labels "subjID", "cond", "prob", "choice", "RT", "feedback". It is not necessary for the columns to be in this particular order, however it is necessary that they be labeled correctly and contain the information below:

subjID

A unique identifier for each subject in the data-set.

cond

Integer value representing the task condition of the given trial (AB == 1, CD == 2, EF == 3).

prob

Float value representing the probability that a correct response (1) is rewarded in the current task condition.

choice

Integer value representing the option chosen on the given trial (1 or 2).

RT

Float value representing the time taken for the response on the given trial.

feedback

Integer value representing the outcome of the given trial (where 'correct' == 1, and 'incorrect' == 0).

*Note: The file may contain other columns of data (e.g. "ReactionTime", "trial_number", etc.), but only the data within the column names listed above will be used during the modeling. As long as the necessary columns mentioned above are present and labeled correctly, there is no need to remove other miscellaneous data columns.

nwarmup is a numerical value that specifies how many MCMC samples should not be stored upon the beginning of each chain. For those familiar with Bayesian methods, this is equivalent to burn-in samples. Due to the nature of the MCMC algorithm, initial values (i.e. where the sampling chains begin) can have a heavy influence on the generated posterior distributions. The nwarmup argument can be set to a high number in order to curb the effects that initial values have on the resulting posteriors.

nchain is a numerical value that specifies how many chains (i.e. independent sampling sequences) should be used to draw samples from the posterior distribution. Since the posteriors are generated from a sampling process, it is good practice to run multiple chains to ensure that a reasonably representative posterior is attained. When the sampling is complete, it is possible to check the multiple chains for convergence by running the following line of code: plot(output, type = "trace"). The trace-plot should resemble a "furry caterpillar".

nthin is a numerical value that specifies the "skipping" behavior of the MCMC sampler, using only every i == nthin samples to generate posterior distributions. By default, nthin is equal to 1, meaning that every sample is used to generate the posterior.

Control Parameters: adapt_delta, stepsize, and max_treedepth are advanced options that give the user more control over Stan's MCMC sampler. It is recommended that only advanced users change the default values, as alterations can profoundly change the sampler's behavior. Refer to 'The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo (Hoffman & Gelman, 2014, Journal of Machine Learning Research)' for more information on the sampler control parameters. One can also refer to 'Section 34.2. HMC Algorithm Parameters' of the Stan User's Guide and Reference Manual, or to the help page for stan for a less technical description of these arguments.

Contributors

Hoyoung Doh <[e-mail]>, Sanghoon Kang <[e-mail]>, Jihyun K. Hur <[e-mail]>

Value

A class "hBayesDM" object modelData with the following components:

model

Character value that is the name of the model (\code"pstRT_rlddm1").

allIndPars

Data.frame containing the summarized parameter values (as specified by indPars) for each subject.

parVals

List object containing the posterior samples over different parameters.

fit

A class stanfit object that contains the fitted Stan model.

rawdata

Data.frame containing the raw data used to fit the model, as specified by the user.

modelRegressor

List object containing the extracted model-based regressors.

References

Frank, M. J., Santamaria, A., O'Reilly, R. C., & Willcutt, E. (2007). Testing computational models of dopamine and noradrenaline dysfunction in attention deficit/hyperactivity disorder. Neuropsychopharmacology, 32(7), 1583-1599.

Frank, M. J., Seeberger, L. C., & O'reilly, R. C. (2004). By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science, 306(5703), 1940-1943.

Pedersen, M. L., Frank, M. J., & Biele, G. (2017). The drift diffusion model as the choice rule in reinforcement learning. Psychonomic bulletin & review, 24(4), 1234-1251.

See Also

We refer users to our in-depth tutorial for an example of using hBayesDM: https://rpubs.com/CCSL/hBayesDM

Examples

## Not run: 
# Run the model with a given data.frame as df
output <- pstRT_rlddm1(
  data = df, niter = 2000, nwarmup = 1000, nchain = 4, ncore = 4)

# Run the model with example data
output <- pstRT_rlddm1(
  data = "example", niter = 2000, nwarmup = 1000, nchain = 4, ncore = 4)

# Visually check convergence of the sampling chains (should look like 'hairy caterpillars')
plot(output, type = "trace")

# Check Rhat values (all Rhat values should be less than or equal to 1.1)
rhat(output)

# Plot the posterior distributions of the hyper-parameters (distributions should be unimodal)
plot(output)

# Show the WAIC and LOOIC model fit estimates
printFit(output)

## End(Not run)