buildBench {SummarizedBenchmark} | R Documentation

## Make SummarizedBenchmark from BenchDesign

### Description

Function to evaluate `BenchDesign` methods on supplied data set to generate a `SummarizedBenchmark`. In addition to the results of applying each method on the data, the returned `SummarizedBenchmark` also includes metadata for the methods in the `colData` of the returned object, metadata for the data in the `rowData`, and session information in the `metadata`.

### Usage

    buildBench(bd, data = NULL, truthCols = NULL, ftCols = NULL,
      sortIDs = FALSE, keepData = FALSE, catchErrors = TRUE,
      parallel = FALSE, BPPARAM = bpparam())

### Arguments

`bd` | `BenchDesign` object.
`data` | Data set to be used for benchmarking, will take priority over data set specified to BenchDesign object. Ignored if NULL. (default = NULL)
`truthCols` | Character vector of column names in data set corresponding to ground truth values for each assay. If specified, column will be added to the `groundTruth` DataFrame of the returned SummarizedBenchmark object. If the BenchDesign includes only a single assay, the same name will be used for the assay. If the BenchDesign includes multiple assays, to map data set columns with assays, the vector must have names corresponding to the assay names specified to the `post` parameter at each `addMethod` call. (default = NULL)
`ftCols` | Vector of character names of columns in data set that should be included as feature data (row data) in the returned SummarizedBenchmark object. (default = NULL)
`sortIDs` | Whether the output of each method should be merged and sorted using IDs. See Details for more information. (default = FALSE)
`keepData` | Whether to store the data as part of the BenchDesign slot of the returned SummarizedBenchmark object. If FALSE, a MD5 hash of the data will be stored with the BenchDesign slot. (default = FALSE)
`catchErrors` | logical whether errors produced by methods during evaluation should be caught and printed as a message without stopping the entire build process. (default = TRUE)
`parallel` | Whether to use parallelization for evaluating each method. Parallel execution is performed using `BiocParallel`. Parameters for parallelization should be specified with `BiocParallel::register` or through the `BPPARAM` parameter. (default = FALSE)
`BPPARAM` | Optional `BiocParallelParam` instance to be used when `parallel` is TRUE. If not specified, the default instance from the parameter registry is used.

### Details

Parallelization is performed across methods. Therefore, there is currently no benefit to specifying more cores than the total number of methods in the `BenchDesign` object.

By default, errors thrown by individual methods in the `BenchDesign` are caught during evaluation and handled in a way that allows `buildBench` to continue running with the other methods. The error is printed as a message, and the corresponding column in the returned `SummarizedBenchmark` object is set to NA. Since many benchmarking experiments can be time and computationally intensive, having to rerun the entire analysis due to a single failed method can be frustrating. Default error catching was included to alleviate these frustrations. However, if this behavior is not desired, setting `catchErrors = FALSE` will turn off error handling.

If `sortIDs` is TRUE, each method must return a named vector or list. The names will be used to align the output of each method in the returned SummarizedBenchmark. Missing values from each method will be set to NA. This can be useful if the different methods return overlapping, but not identical, results. If `truthCols` is also specified, and sorting by IDs is necessary, rather than specifying 'TRUE', specify the string name of a column in the `data` to use to sort the method output to match the order of `truthCols`.

When a method specified in the BenchDesign does not have a postprocessing function specified to 'post', the trivial `base::identity` function is used as the default postprocessing function.

### Value

`SummarizedBenchmark` object.

### Author(s)

Patrick Kimes

### Examples

    ## with toy data.frame
    df <- data.frame(pval = rnorm(100))
    bench <- BenchDesign(data = df)

    ## add methods
    bench <- addMethod(bench, label = "bonf", func = p.adjust,
                       params = rlang::quos(p = pval, method = "bonferroni"))
    bench <- addMethod(bench, label = "BH", func = p.adjust,
                       params = rlang::quos(p = pval, method = "BH"))

    ## evaluate benchmark experiment
    sb <- buildBench(bench)

    ## evaluate benchmark experiment w/ data sepecified
    sb <- buildBench(bench, data = df)

[Package SummarizedBenchmark version 2.0.1 Index]