This report documents the results of a simulation based calibration
(SBC) run for OncoBayes2. TODO
The calibration data presented here has been generated at and with
the OncoBayes git version as:
## Created: 2023-03-01 14:46:01 UTC
## git hash: 6f573afda14f35f01508a91f4c3f9944988125b9
## MD5: 49071dd394a49cf63fd0249d23cf146dThe MD5 hash of the calibration data file presented here must match the above listed MD5:
## /gitlab-runner/builds/53wtLnsi/0/WEBERSE2/OncoBayes2/inst/sbc/calibration.rds
## "49071dd394a49cf63fd0249d23cf146d"Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).
Self-consistency of any Bayesian analysis with a proper prior:
\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]
SBC procedure:
Repeat \(s=1, ..., S\) times:
Sample from the prior \[\tilde{\theta} \sim p(\theta)\]
Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]
Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]
Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]
The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]
The fake data simulation function returns … TODO. Please refer to the
sbc_tools.R and make_reference_rankhist.R R
programs for the implementation details.
The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.
| data_scenario | N | total_divergent | min_ess_bulk | min_ess_tail | max_Rhat | total_large_Rhat | min_lp_ess_bulk | min_lp_ess_tail |
|---|---|---|---|---|---|---|---|---|
| combo2_EX | 10000 | 0 | 470. | 400. | 1.01 | 0 | 328. | 498. |
| combo2_EXNEX | 10000 | 0 | 109. | 50.8 | 1.04 | 0 | 379. | 547. |
| combo3_EXNEX | 10000 | 0 | 9.76 | 17.5 | 1.14 | 1 | 300. | 477. |
| log2bayes_EXNEX | 10000 | 1 | 111. | 257. | 1.02 | 0 | 301. | 537. |
Large Rhat is defined as exceeding \(1.1\).
ESS speed is in units of ESS per second.
| param | statistic | df | p.value |
|---|---|---|---|
| beta_group\[A,I(log(drug_A/1)),intercept\] | 18.899 | 31 | 0.957 |
| beta_group\[A,I(log(drug_A/1)),log_slope\] | 27.238 | 31 | 0.660 |
| beta_group\[B,I(log(drug_A/1)),intercept\] | 23.891 | 31 | 0.815 |
| beta_group\[B,I(log(drug_A/1)),log_slope\] | 23.827 | 31 | 0.818 |
| beta_group\[C,I(log(drug_A/1)),intercept\] | 40.634 | 31 | 0.115 |
| beta_group\[C,I(log(drug_A/1)),log_slope\] | 17.658 | 31 | 0.974 |
| mu_log_beta\[I(log(drug_A/1)),intercept\] | 25.050 | 31 | 0.765 |
| mu_log_beta\[I(log(drug_A/1)),log_slope\] | 18.317 | 31 | 0.965 |
| tau_log_beta\[STRAT,I(log(drug_A/1)),intercept\] | 24.877 | 31 | 0.773 |
| tau_log_beta\[STRAT,I(log(drug_A/1)),log_slope\] | 46.035 | 31 | 0.040 |
| param | statistic | df | p.value |
|---|---|---|---|
| beta_group\[A,I(log(drug_A/1)),intercept\] | 30.918 | 31 | 0.470 |
| beta_group\[A,I(log(drug_A/1)),log_slope\] | 23.059 | 31 | 0.847 |
| beta_group\[A,I(log(drug_B/1)),intercept\] | 32.864 | 31 | 0.376 |
| beta_group\[A,I(log(drug_B/1)),log_slope\] | 19.136 | 31 | 0.953 |
| beta_group\[B,I(log(drug_A/1)),intercept\] | 26.656 | 31 | 0.689 |
| beta_group\[B,I(log(drug_A/1)),log_slope\] | 32.634 | 31 | 0.387 |
| beta_group\[B,I(log(drug_B/1)),intercept\] | 33.632 | 31 | 0.341 |
| beta_group\[B,I(log(drug_B/1)),log_slope\] | 33.632 | 31 | 0.341 |
| beta_group\[C,I(log(drug_A/1)),intercept\] | 25.158 | 31 | 0.761 |
| beta_group\[C,I(log(drug_A/1)),log_slope\] | 37.869 | 31 | 0.184 |
| beta_group\[C,I(log(drug_B/1)),intercept\] | 33.491 | 31 | 0.347 |
| beta_group\[C,I(log(drug_B/1)),log_slope\] | 20.666 | 31 | 0.920 |
| eta_group\[A,I(drug_A/1 \* drug_B/1)\] | 19.642 | 31 | 0.943 |
| eta_group\[B,I(drug_A/1 \* drug_B/1)\] | 18.566 | 31 | 0.962 |
| eta_group\[C,I(drug_A/1 \* drug_B/1)\] | 42.048 | 31 | 0.089 |
| mu_eta\[I(drug_A/1 \* drug_B/1)\] | 22.707 | 31 | 0.860 |
| mu_log_beta\[I(log(drug_A/1)),intercept\] | 23.731 | 31 | 0.821 |
| mu_log_beta\[I(log(drug_A/1)),log_slope\] | 24.128 | 31 | 0.805 |
| mu_log_beta\[I(log(drug_B/1)),intercept\] | 32.122 | 31 | 0.411 |
| mu_log_beta\[I(log(drug_B/1)),log_slope\] | 20.614 | 31 | 0.922 |
| tau_eta\[STRAT,I(drug_A/1 \* drug_B/1)\] | 30.566 | 31 | 0.488 |
| tau_log_beta\[STRAT,I(log(drug_A/1)),intercept\] | 28.832 | 31 | 0.578 |
| tau_log_beta\[STRAT,I(log(drug_A/1)),log_slope\] | 21.581 | 31 | 0.896 |
| tau_log_beta\[STRAT,I(log(drug_B/1)),intercept\] | 32.115 | 31 | 0.411 |
| tau_log_beta\[STRAT,I(log(drug_B/1)),log_slope\] | 14.285 | 31 | 0.996 |
| param | statistic | df | p.value |
|---|---|---|---|
| beta_group\[A,I(log(drug_A/1)),intercept\] | 37.165 | 31 | 0.206 |
| beta_group\[A,I(log(drug_A/1)),log_slope\] | 32.576 | 31 | 0.389 |
| beta_group\[A,I(log(drug_B/1)),intercept\] | 28.416 | 31 | 0.600 |
| beta_group\[A,I(log(drug_B/1)),log_slope\] | 35.789 | 31 | 0.254 |
| beta_group\[B,I(log(drug_A/1)),intercept\] | 29.773 | 31 | 0.529 |
| beta_group\[B,I(log(drug_A/1)),log_slope\] | 36.384 | 31 | 0.232 |
| beta_group\[B,I(log(drug_B/1)),intercept\] | 23.514 | 31 | 0.830 |
| beta_group\[B,I(log(drug_B/1)),log_slope\] | 34.784 | 31 | 0.292 |
| beta_group\[C,I(log(drug_A/1)),intercept\] | 31.507 | 31 | 0.441 |
| beta_group\[C,I(log(drug_A/1)),log_slope\] | 30.125 | 31 | 0.511 |
| beta_group\[C,I(log(drug_B/1)),intercept\] | 26.259 | 31 | 0.709 |
| beta_group\[C,I(log(drug_B/1)),log_slope\] | 26.822 | 31 | 0.681 |
| eta_group\[A,I(drug_A/1 \* drug_B/1)\] | 24.499 | 31 | 0.790 |
| eta_group\[B,I(drug_A/1 \* drug_B/1)\] | 34.701 | 31 | 0.296 |
| eta_group\[C,I(drug_A/1 \* drug_B/1)\] | 30.490 | 31 | 0.492 |
| mu_eta\[I(drug_A/1 \* drug_B/1)\] | 25.286 | 31 | 0.755 |
| mu_log_beta\[I(log(drug_A/1)),intercept\] | 31.450 | 31 | 0.444 |
| mu_log_beta\[I(log(drug_A/1)),log_slope\] | 20.979 | 31 | 0.912 |
| mu_log_beta\[I(log(drug_B/1)),intercept\] | 39.392 | 31 | 0.143 |
| mu_log_beta\[I(log(drug_B/1)),log_slope\] | 25.779 | 31 | 0.732 |
| tau_eta\[STRAT,I(drug_A/1 \* drug_B/1)\] | 23.539 | 31 | 0.829 |
| tau_log_beta\[STRAT,I(log(drug_A/1)),intercept\] | 23.949 | 31 | 0.813 |
| tau_log_beta\[STRAT,I(log(drug_A/1)),log_slope\] | 27.059 | 31 | 0.669 |
| tau_log_beta\[STRAT,I(log(drug_B/1)),intercept\] | 21.715 | 31 | 0.892 |
| tau_log_beta\[STRAT,I(log(drug_B/1)),log_slope\] | 40.896 | 31 | 0.110 |
| param | statistic | df | p.value |
|---|---|---|---|
| beta_group\[A,I(log(drug_A/1)),intercept\] | 23.021 | 31 | 0.848 |
| beta_group\[A,I(log(drug_A/1)),log_slope\] | 25.594 | 31 | 0.741 |
| beta_group\[A,I(log(drug_B/1)),intercept\] | 23.040 | 31 | 0.848 |
| beta_group\[A,I(log(drug_B/1)),log_slope\] | 15.923 | 31 | 0.988 |
| beta_group\[A,I(log(drug_C/1)),intercept\] | 25.043 | 31 | 0.766 |
| beta_group\[A,I(log(drug_C/1)),log_slope\] | 35.789 | 31 | 0.254 |
| beta_group\[B,I(log(drug_A/1)),intercept\] | 25.805 | 31 | 0.731 |
| beta_group\[B,I(log(drug_A/1)),log_slope\] | 27.078 | 31 | 0.668 |
| beta_group\[B,I(log(drug_B/1)),intercept\] | 28.198 | 31 | 0.611 |
| beta_group\[B,I(log(drug_B/1)),log_slope\] | 31.520 | 31 | 0.440 |
| beta_group\[B,I(log(drug_C/1)),intercept\] | 43.942 | 31 | 0.062 |
| beta_group\[B,I(log(drug_C/1)),log_slope\] | 41.702 | 31 | 0.095 |
| beta_group\[C,I(log(drug_A/1)),intercept\] | 27.469 | 31 | 0.648 |
| beta_group\[C,I(log(drug_A/1)),log_slope\] | 41.958 | 31 | 0.091 |
| beta_group\[C,I(log(drug_B/1)),intercept\] | 28.358 | 31 | 0.603 |
| beta_group\[C,I(log(drug_B/1)),log_slope\] | 18.458 | 31 | 0.963 |
| beta_group\[C,I(log(drug_C/1)),intercept\] | 34.509 | 31 | 0.304 |
| beta_group\[C,I(log(drug_C/1)),log_slope\] | 36.122 | 31 | 0.242 |
| eta_group\[A,I(drug_A/1 \* drug_B/1 \* drug_C/1)\] | 19.270 | 31 | 0.950 |
| eta_group\[A,I(drug_A/1 \* drug_B/1)\] | 40.749 | 31 | 0.113 |
| eta_group\[A,I(drug_A/1 \* drug_C/1)\] | 26.010 | 31 | 0.721 |
| eta_group\[A,I(drug_B/1 \* drug_C/1)\] | 29.798 | 31 | 0.528 |
| eta_group\[B,I(drug_A/1 \* drug_B/1 \* drug_C/1)\] | 27.693 | 31 | 0.637 |
| eta_group\[B,I(drug_A/1 \* drug_B/1)\] | 20.461 | 31 | 0.925 |
| eta_group\[B,I(drug_A/1 \* drug_C/1)\] | 42.682 | 31 | 0.079 |
| eta_group\[B,I(drug_B/1 \* drug_C/1)\] | 32.614 | 31 | 0.387 |
| eta_group\[C,I(drug_A/1 \* drug_B/1 \* drug_C/1)\] | 24.877 | 31 | 0.773 |
| eta_group\[C,I(drug_A/1 \* drug_B/1)\] | 28.090 | 31 | 0.617 |
| eta_group\[C,I(drug_A/1 \* drug_C/1)\] | 15.795 | 31 | 0.989 |
| eta_group\[C,I(drug_B/1 \* drug_C/1)\] | 29.939 | 31 | 0.520 |
| mu_eta\[I(drug_A/1 \* drug_B/1 \* drug_C/1)\] | 32.333 | 31 | 0.401 |
| mu_eta\[I(drug_A/1 \* drug_B/1)\] | 43.725 | 31 | 0.064 |
| mu_eta\[I(drug_A/1 \* drug_C/1)\] | 23.962 | 31 | 0.812 |
| mu_eta\[I(drug_B/1 \* drug_C/1)\] | 37.075 | 31 | 0.209 |
| mu_log_beta\[I(log(drug_A/1)),intercept\] | 23.136 | 31 | 0.844 |
| mu_log_beta\[I(log(drug_A/1)),log_slope\] | 30.368 | 31 | 0.498 |
| mu_log_beta\[I(log(drug_B/1)),intercept\] | 25.907 | 31 | 0.726 |
| mu_log_beta\[I(log(drug_B/1)),log_slope\] | 38.534 | 31 | 0.166 |
| mu_log_beta\[I(log(drug_C/1)),intercept\] | 31.411 | 31 | 0.446 |
| mu_log_beta\[I(log(drug_C/1)),log_slope\] | 31.219 | 31 | 0.455 |
| tau_eta\[STRAT,I(drug_A/1 \* drug_B/1 \* drug_C/1)\] | 27.706 | 31 | 0.636 |
| tau_eta\[STRAT,I(drug_A/1 \* drug_B/1)\] | 23.859 | 31 | 0.816 |
| tau_eta\[STRAT,I(drug_A/1 \* drug_C/1)\] | 67.322 | 31 | 0.000 |
| tau_eta\[STRAT,I(drug_B/1 \* drug_C/1)\] | 34.637 | 31 | 0.298 |
| tau_log_beta\[STRAT,I(log(drug_A/1)),intercept\] | 33.478 | 31 | 0.348 |
| tau_log_beta\[STRAT,I(log(drug_A/1)),log_slope\] | 33.056 | 31 | 0.367 |
| tau_log_beta\[STRAT,I(log(drug_B/1)),intercept\] | 26.202 | 31 | 0.712 |
| tau_log_beta\[STRAT,I(log(drug_B/1)),log_slope\] | 31.059 | 31 | 0.463 |
| tau_log_beta\[STRAT,I(log(drug_C/1)),intercept\] | 50.170 | 31 | 0.016 |
| tau_log_beta\[STRAT,I(log(drug_C/1)),log_slope\] | 46.214 | 31 | 0.039 |
## R version 4.1.0 (2021-05-18)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 20.04.5 LTS
##
## Matrix products: default
## BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0
##
## locale:
## [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
## [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
## [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
## [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
## [9] LC_ADDRESS=C LC_TELEPHONE=C
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
##
## attached base packages:
## [1] tools stats graphics grDevices utils datasets methods
## [8] base
##
## other attached packages:
## [1] ggplot2_3.3.5 broom_0.7.9 tidyr_1.1.3 dplyr_1.0.8
## [5] assertthat_0.2.1 knitr_1.33 here_1.0.1
##
## loaded via a namespace (and not attached):
## [1] pillar_1.6.2 compiler_4.1.0 highr_0.9 digest_0.6.29
## [5] evaluate_0.14 lifecycle_1.0.1 tibble_3.1.3 gtable_0.3.0
## [9] pkgconfig_2.0.3 rlang_1.0.6 cli_3.1.1 DBI_1.1.2
## [13] yaml_2.2.1 xfun_0.25 fastmap_1.1.0 withr_2.4.3
## [17] stringr_1.4.0 generics_0.1.0 vctrs_0.5.2 rprojroot_2.0.2
## [21] grid_4.1.0 tidyselect_1.1.1 glue_1.6.1 R6_2.5.1
## [25] fansi_0.5.0 rmarkdown_2.11 purrr_0.3.4 magrittr_2.0.1
## [29] backports_1.2.1 scales_1.1.1 ellipsis_0.3.2 htmltools_0.5.2
## [33] colorspace_2.0-2 utf8_1.2.2 stringi_1.7.3 munsell_0.5.0
## [37] crayon_1.4.2