using_time_models

Victor Navarro

Time models in calmr

Version 0.5 of calmr introduced its first time-based model, ANCCR (Jeong et al., 2022), and with it, I wrote several additional tools for future models.

Changes to trial-based models

The biggest change in calmr version 0.5 is the use of the “>” character and its effect on trial-based models. Before, the “>” character was used to specify a single split within a trial. For example, “A>(US)” would encode the typical situation in which stimulus A is followed by the US. This was used to mimic the traditional situation in which we expect an organism to start (conditionally) responding before the US is delivered. And so, all trial-based models had two steps within each trial: an expectation step in which the first half of the trial retrieved absent stimuli, and a learning step, in which all stimuli in the trial were associated with each other. The first pass is no more (so start throwing extinction trials, or better yet, probe trials to test your associations).

Specifying a design for time-based models

The designs for time-based models are nearly identical to those for trial-based models. However, clever use of the “>” character will enrich the parameter list. Let’s specify a serial feature discrimination experiment:

library(calmr)
fpfn <- data.frame(
  group = c("FP", "FN"),
  phase1 = c("100F>T>(US)/100T", "100F>T/100T>(US)"),
  r1 = c(TRUE, TRUE)
)
fpfn
#>   group           phase1   r1
#> 1    FP 100F>T>(US)/100T TRUE
#> 2    FN 100F>T/100T>(US) TRUE

And now let’s get the parameters for the ANCCR model.

pars <- get_parameters(fpfn, model = "ANCCR")
# increase learning rates
pars$alpha_reward <- 0.8
pars$alpha <- 0.08
# increase sampling interval to speed up the model
pars$sampling_interval <- 5
pars
#> $reward_magnitude
#>  F  T US 
#>  1  1  1 
#> 
#> $betas
#>  F  T US 
#>  1  1  1 
#> 
#> $cost
#> [1] 0
#> 
#> $temperature
#> [1] 1
#> 
#> $threshold
#> [1] 0.6
#> 
#> $k
#> [1] 1
#> 
#> $w
#> [1] 0.5
#> 
#> $minimum_rate
#> [1] 0.001
#> 
#> $sampling_interval
#> [1] 5
#> 
#> $use_exact_mean
#> [1] 0
#> 
#> $use_exponential
#> [1] 1
#> 
#> $t_ratio
#> [1] 1.2
#> 
#> $t_constant
#> [1] NA
#> 
#> $t_jitter
#> [1] 0.1
#> 
#> $alpha
#> [1] 0.08
#> 
#> $alpha_reward
#> [1] 0.8
#> 
#> $use_timed_alpha
#> [1] 0
#> 
#> $alpha_exponent
#> [1] 1
#> 
#> $alpha_init
#> [1] 1
#> 
#> $alpha_min
#> [1] 0
#> 
#> $add_beta
#> [1] 0
#> 
#> $post_trial_delay
#> F>T>(US)        T      F>T   T>(US) 
#>        1        1        1        1 
#> 
#> $mean_ITI
#> F>T>(US)        T      F>T   T>(US) 
#>       30       30       30       30 
#> 
#> $max_ITI
#> F>T>(US)        T      F>T   T>(US) 
#>       90       90       90       90 
#> 
#> $transition_delay
#> $transition_delay$`F>T>(US)`
#>    F>T T>(US) 
#>      1      1 
#> 
#> $transition_delay$`F>T`
#> F>T 
#>   1 
#> 
#> $transition_delay$`T>(US)`
#> T>(US) 
#>      1

The ANCCR model has plenty of parameters, yet nearly half of the parameters in the above list correspond to parameters we will use to create the experience that the model will receive. I leave the explanation of model-based parameters for the future. For now, it will suffice to note that we can control the timing between trials (and their events) using things like post_trial_delay, mean_ITI, transition_delay, etc.

Let’s make the model’s experience and look at the first 20 entries.

experiment <- make_experiment(fpfn,
  parameters = pars,
  model = "ANCCR"
)
head(experiences(experiment)[[1]], 20)
#>    model group  phase tp       tn is_test block_size trial stimulus       time reward_mag
#> 1  ANCCR    FP phase1  1 F>T>(US)   FALSE          2     1        F   2.476256          1
#> 2  ANCCR    FP phase1  1 F>T>(US)   FALSE          2     1        T   3.476256          1
#> 3  ANCCR    FP phase1  1 F>T>(US)   FALSE          2     1       US   4.476256          1
#> 4  ANCCR    FP phase1  2        T   FALSE          2     2        T  32.700505          1
#> 5  ANCCR    FP phase1  2        T   FALSE          2     3        T  47.490325          1
#> 6  ANCCR    FP phase1  1 F>T>(US)   FALSE          2     4        F  57.711270          1
#> 7  ANCCR    FP phase1  1 F>T>(US)   FALSE          2     4        T  58.711270          1
#> 8  ANCCR    FP phase1  1 F>T>(US)   FALSE          2     4       US  59.711270          1
#> 9  ANCCR    FP phase1  2        T   FALSE          2     5        T  61.010581          1
#> 10 ANCCR    FP phase1  1 F>T>(US)   FALSE          2     6        F  83.211286          1
#> 11 ANCCR    FP phase1  1 F>T>(US)   FALSE          2     6        T  84.211286          1
#> 12 ANCCR    FP phase1  1 F>T>(US)   FALSE          2     6       US  85.211286          1
#> 13 ANCCR    FP phase1  2        T   FALSE          2     7        T 114.709336          1
#> 14 ANCCR    FP phase1  1 F>T>(US)   FALSE          2     8        F 141.713618          1
#> 15 ANCCR    FP phase1  1 F>T>(US)   FALSE          2     8        T 142.713618          1
#> 16 ANCCR    FP phase1  1 F>T>(US)   FALSE          2     8       US 143.713618          1
#> 17 ANCCR    FP phase1  1 F>T>(US)   FALSE          2     9        F 156.609796          1
#> 18 ANCCR    FP phase1  1 F>T>(US)   FALSE          2     9        T 157.609796          1
#> 19 ANCCR    FP phase1  1 F>T>(US)   FALSE          2     9       US 158.609796          1
#> 20 ANCCR    FP phase1  2        T   FALSE          2    10        T 165.795586          1

As you can see above, there are several rows per trial, each specifying a different stimulus. Time-based models like ANCCR run over a time log because they make ample use of the time difference between events.

Let’s run the model and see some plots.

experiment <- run_experiment(experiment)
# Action values
patch_plots(plot(experiment, type = "qs"))

# ANCCR
patch_plots(plot(experiment, type = "anccrs"))

# Dopamine transients
patch_plots(plot(experiment, type = "das"))

And that’s it! Easy, right?

References

Jeong, H., Taylor, A., Floeder, J. R., Lohmann, M., Mihalas, S., Wu, B., Zhou, M., Burke, D. A., & Namboodiri, V. M. K. (2022). Mesolimbic dopamine release conveys causal associations. Science (New York, N.Y.), 378, eabq6740. https://doi.org/10.1126/science.abq6740