This report documents the results of a simulation based calibration (SBC) run for RBesT. The calibration data will be generated whenever relevant changes to the gMAP function were made. The calibration runs are performed for typical use cases of gMAP. These include the three likelihoods (binomial, gaussian & Poisson), a sparse (\(2\) trials) and dense (\(10\) trials) data situation and finally a run with a very/less conservative prior choice for between-trial heterogeniety parameter.

The calibration data presented here has been generated at and with the RBesT git version as:

## Created:  2022-07-25 15:43:29 UTC
## git hash: a9012f0d7bf708d5f2998048157da97f3921e166
## MD5:      e085f208d722d6ce54f5e7cbb5bafe62

The MD5 hash of the calibration data file presented here must match the above listed MD5:

## /Users/weberse2/work/BesT/inst/sbc/calibration.rds 
##                 "e085f208d722d6ce54f5e7cbb5bafe62"

Introduction

Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).

Self-consistency of any Bayesian analysis with a proper prior:

\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]

SBC procedure:

Repeat \(s=1, ..., S\) times:

Sample from the prior \[\tilde{\theta} \sim p(\theta)\]
Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]
Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]
Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]

The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]

Hierarchical intercept only (random-effects intercept) model for binomial, gaussian and Poisson likelihood

Likelihood:

Binary \[y_i|\theta_{j} \sim \mbox{Bernoulli}(\theta_j), \] \[g(\theta) = \mbox{logit}(\theta)\]
Normal \[y_i|\theta_{j} \sim \mbox{Normal}(\theta_j, \sigma^2), \] \[g(\theta) = \theta\]
Poisson \[y_i|\theta_{j} \sim \mbox{Poisson}(\theta_j), \] \[g(\theta) = \log(\theta)\]

Hierarchical prior:

\[ g(\theta_j)|\mu,\tau \sim \mbox{Normal}(\mu, \tau^2)\]

\[\mu \sim \mbox{Normal}(m_\mu, s^2_\mu)\] \[\tau \sim \mbox{Normal}^+(0, s^2_\tau)\]

The fake data simulation function returns for binomial and Poisson data the sum of the responses while for normal the mean summary is used. Please refer to the sbc_tools.R and make_reference_rankhist.R R programs for the implementation details.

The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.

SBC results

Sampler Diagnostics Overview

family	data_scenario	sd_tau	N	total_divergent	total_divergent_sim_fraction	min_ess	max_Rhat	total_large_Rhat	min_lp_ess_bulk	min_lp_ess_tail
binomial	dense	0.5	10000	41	0.004	173	1.013	0	196	169
binomial	dense	1.0	10000	65	0.005	85	1.024	0	200	171
binomial	sparse	0.5	10000	0	0.000	343	1.013	0	153	163
binomial	sparse	1.0	10000	7	0.001	251	1.018	0	189	130
gaussian	dense	0.5	10000	87	0.008	221	1.023	0	62	173
gaussian	dense	1.0	10000	93	0.007	73	1.027	0	149	168
gaussian	sparse	0.5	10000	739	0.036	8	1.265	1	33	133
gaussian	sparse	1.0	10000	1282	0.062	12	1.236	1	105	119
poisson	dense	0.5	10000	112	0.009	126	1.023	0	81	149
poisson	dense	1.0	10000	145	0.013	29	1.118	0	84	230
poisson	sparse	0.5	10000	1536	0.048	6	1.439	1	127	120
poisson	sparse	1.0	10000	1902	0.063	3	1.733	1	45	51

Note: Large Rhat is defined as exceeding 1.2.

Summary Statistics

\(\chi^2\) Statistic, \(\mu\)

	data_scenario	likelihood	sd_tau	parameter	statistic	df	p.value
1	dense	binomial	0.5	mu	75.341	63	0.137
13	dense	binomial	1	mu	66.074	63	0.371
25	dense	gaussian	0.5	mu	65.485	63	0.391
37	dense	gaussian	1	mu	58.202	63	0.648
49	dense	poisson	0.5	mu	63.258	63	0.467
61	dense	poisson	1	mu	88.410	63	0.019
73	sparse	binomial	0.5	mu	63.322	63	0.465
78	sparse	binomial	1	mu	72.000	63	0.205
83	sparse	gaussian	0.5	mu	47.373	63	0.929
88	sparse	gaussian	1	mu	80.282	63	0.070
93	sparse	poisson	0.5	mu	56.819	63	0.695
98	sparse	poisson	1	mu	76.698	63	0.115

\(\chi^2\) Statistic, \(\tau\)

	data_scenario	likelihood	sd_tau	parameter	statistic	df	p.value
2	dense	binomial	0.5	tau	74.074	63	0.160
14	dense	binomial	1	tau	69.402	63	0.271
26	dense	gaussian	0.5	tau	99.622	63	0.002
38	dense	gaussian	1	tau	64.922	63	0.410
50	dense	poisson	0.5	tau	74.368	63	0.155
62	dense	poisson	1	tau	51.238	63	0.856
74	sparse	binomial	0.5	tau	51.546	63	0.848
79	sparse	binomial	1	tau	58.227	63	0.647
84	sparse	gaussian	0.5	tau	50.432	63	0.874
89	sparse	gaussian	1	tau	54.080	63	0.781
94	sparse	poisson	0.5	tau	44.147	63	0.966
99	sparse	poisson	1	tau	62.515	63	0.494

\(\chi^2\) Statistic, group estimates \(\theta\)

	data_scenario	likelihood	sd_tau	parameter	statistic	df	p.value
3	dense	binomial	0.5	theta\[1\]	79.846	63	0.075
4	dense	binomial	0.5	theta\[10\]	43.597	63	0.970
5	dense	binomial	0.5	theta\[2\]	60.954	63	0.550
6	dense	binomial	0.5	theta\[3\]	54.797	63	0.760
7	dense	binomial	0.5	theta\[4\]	57.728	63	0.664
8	dense	binomial	0.5	theta\[5\]	68.134	63	0.307
9	dense	binomial	0.5	theta\[6\]	42.816	63	0.976
10	dense	binomial	0.5	theta\[7\]	47.334	63	0.929
11	dense	binomial	0.5	theta\[8\]	55.027	63	0.753
12	dense	binomial	0.5	theta\[9\]	63.578	63	0.456
15	dense	binomial	1	theta\[1\]	85.222	63	0.033
16	dense	binomial	1	theta\[10\]	46.490	63	0.941
17	dense	binomial	1	theta\[2\]	67.264	63	0.333
18	dense	binomial	1	theta\[3\]	67.149	63	0.337
19	dense	binomial	1	theta\[4\]	49.766	63	0.887
20	dense	binomial	1	theta\[5\]	76.941	63	0.111
21	dense	binomial	1	theta\[6\]	47.782	63	0.923
22	dense	binomial	1	theta\[7\]	55.155	63	0.749
23	dense	binomial	1	theta\[8\]	53.210	63	0.806
24	dense	binomial	1	theta\[9\]	55.782	63	0.729
27	dense	gaussian	0.5	theta\[1\]	51.456	63	0.850
28	dense	gaussian	0.5	theta\[10\]	68.442	63	0.298
29	dense	gaussian	0.5	theta\[2\]	84.006	63	0.040
30	dense	gaussian	0.5	theta\[3\]	68.736	63	0.289
31	dense	gaussian	0.5	theta\[4\]	63.104	63	0.473
32	dense	gaussian	0.5	theta\[5\]	57.690	63	0.665
33	dense	gaussian	0.5	theta\[6\]	53.146	63	0.807
34	dense	gaussian	0.5	theta\[7\]	67.366	63	0.330
35	dense	gaussian	0.5	theta\[8\]	63.974	63	0.442
36	dense	gaussian	0.5	theta\[9\]	61.517	63	0.529
39	dense	gaussian	1	theta\[1\]	57.574	63	0.669
40	dense	gaussian	1	theta\[10\]	62.003	63	0.512
41	dense	gaussian	1	theta\[2\]	61.786	63	0.520
42	dense	gaussian	1	theta\[3\]	83.008	63	0.046
43	dense	gaussian	1	theta\[4\]	65.254	63	0.398
44	dense	gaussian	1	theta\[5\]	70.733	63	0.235
45	dense	gaussian	1	theta\[6\]	61.402	63	0.533
46	dense	gaussian	1	theta\[7\]	86.822	63	0.025
47	dense	gaussian	1	theta\[8\]	77.888	63	0.098
48	dense	gaussian	1	theta\[9\]	77.862	63	0.098
51	dense	poisson	0.5	theta\[1\]	50.163	63	0.879
52	dense	poisson	0.5	theta\[10\]	59.251	63	0.611
53	dense	poisson	0.5	theta\[2\]	51.110	63	0.859
54	dense	poisson	0.5	theta\[3\]	59.661	63	0.596
55	dense	poisson	0.5	theta\[4\]	46.925	63	0.935
56	dense	poisson	0.5	theta\[5\]	52.096	63	0.835
57	dense	poisson	0.5	theta\[6\]	55.846	63	0.727
58	dense	poisson	0.5	theta\[7\]	55.872	63	0.726
59	dense	poisson	0.5	theta\[8\]	76.442	63	0.119
60	dense	poisson	0.5	theta\[9\]	55.782	63	0.729
63	dense	poisson	1	theta\[1\]	51.123	63	0.858
64	dense	poisson	1	theta\[10\]	74.752	63	0.148
65	dense	poisson	1	theta\[2\]	51.725	63	0.844
66	dense	poisson	1	theta\[3\]	49.549	63	0.892
67	dense	poisson	1	theta\[4\]	61.325	63	0.536
68	dense	poisson	1	theta\[5\]	67.392	63	0.329
69	dense	poisson	1	theta\[6\]	57.702	63	0.665
70	dense	poisson	1	theta\[7\]	60.314	63	0.573
71	dense	poisson	1	theta\[8\]	53.811	63	0.789
72	dense	poisson	1	theta\[9\]	72.371	63	0.196
75	sparse	binomial	0.5	theta\[1\]	46.682	63	0.938
76	sparse	binomial	0.5	theta\[2\]	57.536	63	0.671
77	sparse	binomial	0.5	theta\[3\]	69.709	63	0.262
80	sparse	binomial	1	theta\[1\]	60.992	63	0.548
81	sparse	binomial	1	theta\[2\]	56.128	63	0.718
82	sparse	binomial	1	theta\[3\]	66.842	63	0.347
85	sparse	gaussian	0.5	theta\[1\]	61.658	63	0.524
86	sparse	gaussian	0.5	theta\[2\]	85.184	63	0.033
87	sparse	gaussian	0.5	theta\[3\]	62.195	63	0.505
90	sparse	gaussian	1	theta\[1\]	49.382	63	0.895
91	sparse	gaussian	1	theta\[2\]	63.744	63	0.450
92	sparse	gaussian	1	theta\[3\]	57.062	63	0.687
95	sparse	poisson	0.5	theta\[1\]	63.117	63	0.472
96	sparse	poisson	0.5	theta\[2\]	59.430	63	0.604
97	sparse	poisson	0.5	theta\[3\]	64.858	63	0.412
100	sparse	poisson	1	theta\[1\]	70.259	63	0.248
101	sparse	poisson	1	theta\[2\]	69.491	63	0.268
102	sparse	poisson	1	theta\[3\]	66.829	63	0.347

Session Info

## R version 4.1.0 (2021-05-18)
## Platform: x86_64-apple-darwin17.0 (64-bit)
## Running under: macOS Big Sur 10.16
## 
## Matrix products: default
## BLAS:   /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## attached base packages:
## [1] tools     stats     graphics  grDevices datasets  utils     methods  
## [8] base     
## 
## other attached packages:
##  [1] purrr_0.3.4          rstan_2.21.2         StanHeaders_2.21.0-7
##  [4] here_1.0.1           ggplot2_3.3.5        broom_0.7.9         
##  [7] tidyr_1.1.3          dplyr_1.0.7          assertthat_0.2.1    
## [10] knitr_1.41          
## 
## loaded via a namespace (and not attached):
##  [1] tidyselect_1.1.1   xfun_0.36          V8_3.4.2           colorspace_2.0-2  
##  [5] vctrs_0.3.8        generics_0.1.0     htmltools_0.5.1.1  stats4_4.1.0      
##  [9] loo_2.4.1          yaml_2.2.1         utf8_1.2.2         rlang_0.4.11      
## [13] pkgbuild_1.2.0     pillar_1.6.2       glue_1.4.2         withr_2.4.2       
## [17] DBI_1.1.1          matrixStats_0.60.1 lifecycle_1.0.0    stringr_1.4.0     
## [21] munsell_0.5.0      gtable_0.3.0       codetools_0.2-18   evaluate_0.20     
## [25] inline_0.3.19      callr_3.7.0        ps_1.6.0           parallel_4.1.0    
## [29] curl_4.3.2         fansi_0.5.0        highr_0.9          Rcpp_1.0.7        
## [33] scales_1.1.1       backports_1.2.1    RcppParallel_5.1.4 jsonlite_1.7.2    
## [37] gridExtra_2.3      digest_0.6.27      stringi_1.7.3      processx_3.5.2    
## [41] grid_4.1.0         rprojroot_2.0.2    cli_3.0.1          magrittr_2.0.1    
## [45] tibble_3.1.3       crayon_1.4.1       pkgconfig_2.0.3    ellipsis_0.3.2    
## [49] prettyunits_1.1.1  rmarkdown_2.19     R6_2.5.1           compiler_4.1.0