# ReUseData: Reusable and Reproducible Data Management

Qian Liu, Qiang Hu, Song Liu, Alan Hutson, Martin Morgan1

1Roswell Park Comprehensive Cancer Center

#### 2023-07-31

`ReUseData` provides functionalities to construct workflow-based data recipes for fully tracked and reproducible data processing. Evaluation of data recipes generates curated data resources in their generic formats (e.g., VCF, bed), as well as a YAML manifest file recording the recipe parameters, data annotations, and data file paths for subsequent reuse. The datasets are locally cached using a database infrastructure, where updating and searching of specific data is made easy.

The data reusability is assured through cloud hosting and enhanced interoperability with downstream software tools or analysis workflows. The workflow strategy enables cross platform reproducibility of curated data resources.

# 1 Installation

1. Install the package from Bioconductor.

    if (!requireNamespace("BiocManager", quietly = TRUE))
        install.packages("BiocManager")
    BiocManager::install("ReUseData")

Use the development version:

    BiocManager::install("ReUseData", version = "devel")

2. Load the package and other packages used in this vignette into the R session.

    suppressPackageStartupMessages(library(Rcwl))
    library(ReUseData)

# 2 `ReUseData` core functions for data management

Here we introduce the core functions of `ReUseData` for data management and reuse: `getData` for reproducible data generation, `dataUpdate` for syncing and updating data cache, and `dataSearch` for multi-keywords searching of dataset of interest.

## 2.1 Data generation

First, we can construct data recipes by transforming shell or other ad hoc data preprocessing scripts into workflow-based data recipes. Some prebuilt data recipes for public data resources (e.g., downloading, unzipping and indexing) are available for direct use through `recipeSearch` and `recipeLoad` functions. Then we will assign values to the input parameters and evaluate the recipe to generate data of interest.

    recipeUpdate()
    #> Updating recipes...
    #>
    #> recipeHub with 15 records
    #> cache path:  /Users/biocbuild/Library/Caches/org.R-project.R/R/ReUseDataRecipe
    #> # recipeSearch() to query specific recipes using multipe keywords
    #> # recipeUpdate() to update the local recipe cache
    #>
    #>            name
    #>   BFC916 | STAR_index
    #>   BFC917 | bowtie2_index
    #>   BFC918 | echo_out
    #>   BFC919 | ensembl_liftover
    #>   BFC920 | gcp_broad_gatk_hg19
    #>   ...      ...
    #>   BFC926 | gencode_transcripts
    #>   BFC927 | hisat2_index
    #>   BFC928 | reference_genome
    #>   BFC929 | salmon_index
    #>   BFC930 | ucsc_database
    recipeSearch("echo")
    #> recipeHub with 1 records
    #> cache path:  /Users/biocbuild/Library/Caches/org.R-project.R/R/ReUseDataRecipe
    #> # recipeSearch() to query specific recipes using multipe keywords
    #> # recipeUpdate() to update the local recipe cache
    #>
    #>            name
    #>   BFC918 | echo_out
    echo_out <- recipeLoad("echo_out")
    #> Note: you need to assign a name for the recipe: rcpName <- recipeLoad('xx')
    #> Data recipe loaded!
    #> Use inputs() to check required input parameters before evaluation.
    #> Check here: https://rcwl.org/dataRecipes/echo_out.html
    #> for user instructions (e.g., eligible input values, data source, etc.)
    inputs(echo_out)
    #> inputs:
    #>   input (input)  (string):
    #>   outfile (outfile)  (string):

Users can then assign values to the input parameters, and evaluate the recipe (`getData`) to generate data of interest. Users need to specify an output directory for all files (desired data file, intermediate files that are internally generated as workflow scripts or annotation files). Detailed notes for the data is encouraged which will be used for keywords matching for later data search.

    echo_out$input <- "Hello World!"
    echo_out$outfile <- "outfile"
    outdir <- file.path(tempdir(), "SharedData")
    res <- getData(echo_out,
                   outdir = outdir,
                   notes = c("echo", "hello", "world", "txt"))
    #> }[1;30mINFO[0m Final process status is success

The file path to newly generated dataset can be easily retrieved. It can also be retrieved using `dataSearch()` functions with multiple keywords. Before that, `dataUpdate()` needs to be done.

    res$output
    #> [1] "/tmp/RtmpDamyzk/SharedData/outfile.txt"

There are some automatically generated files to help track the data recipe evaluation, including `*.sh` to record the original shell script, `*.cwl` file as the official workflow script which was internally submitted for data recipe evaluation, `*.yml` file as part of CWL workflow evaluation, which also record data annotations, and `*.md5` checksum file to check/verify the integrity of generated data file.

    list.files(outdir, pattern = "echo")
    #> [1] "echo_out_Hello_World!_outfile.cwl" "echo_out_Hello_World!_outfile.md5"
    #> [3] "echo_out_Hello_World!_outfile.sh"  "echo_out_Hello_World!_outfile.yml"

The `*.yml` file contains information about recipe input parameters, the file path to output file, the notes for the dataset, and auto-added date for data generation time. A later data search using `dataSearch()` will refer to this file for keywords match.

    readLines(res$yml)
    #> [1] "input: Hello World!"
    #> [2] "outfile: outfile"
    #> [3] "# output: /tmp/RtmpDamyzk/SharedData/outfile.txt"
    #> [4] "# notes: echo hello world txt"
    #> [5] "# date: 2023-07-31 14:47:57.88844"

## 2.2 Data caching, updating and searching

`dataUpdate()` creates (if first time use), syncs and updates the local cache for curated datasets. It finds and reads all the `.yml` files recursively in the provided data folder, creates a cache record for each dataset that is associated (including newly generated ones with `getData()`), and updates the local cache for later data searching and reuse.

IMPORTANT: It is recommended that users create a specified folder for data archival (e.g., `file/path/to/SharedData`) that other group members have access to, and use sub-folders for different kinds of datasets (e.g., those generated from same recipe).

    (dh <- dataUpdate(dir = outdir))
    #>
    #> Updating data record...
    #> outfile.txt added
    #> Warning in dm$tag[idx] <- value: number of items to replace is not a multiple
    #> of replacement length
    #> dataHub with 1 records
    #> cache path:  /Users/biocbuild/Library/Caches/org.R-project.R/R/ReUseData
    #> # dataUpdate() to update the local data cache
    #> # dataSearch() to query a specific dataset
    #> # Additional information can be retrieved using:
    #> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
    #>
    #>              name        Path
    #>   BFC12535 | outfile.txt /tmp/RtmpDamyzk/SharedData/outfile.txt

`dataUpdate` and `dataSearch` return a `dataHub` object with a list of all available or matching datasets.

One can subset the list with `[` and use getter functions to retrieve the annotation information about the data, e.g., data names, parameters values to the recipe, notes, tags, and the corresponding yaml file.

    dh[1]
    #> dataHub with 1 records
    #> cache path:  /Users/biocbuild/Library/Caches/org.R-project.R/R/ReUseData
    #> # dataUpdate() to update the local data cache
    #> # dataSearch() to query a specific dataset
    #> # Additional information can be retrieved using:
    #> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
    #>
    #>              name        Path
    #>   BFC12535 | outfile.txt /tmp/RtmpDamyzk/SharedData/outfile.txt
    ## dh["BFC1"]
    dh[dataNames(dh) == "outfile.txt"]
    #> dataHub with 1 records
    #> cache path:  /Users/biocbuild/Library/Caches/org.R-project.R/R/ReUseData
    #> # dataUpdate() to update the local data cache
    #> # dataSearch() to query a specific dataset
    #> # Additional information can be retrieved using:
    #> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
    #>
    #>              name        Path
    #>   BFC12535 | outfile.txt /tmp/RtmpDamyzk/SharedData/outfile.txt
    dataNames(dh)
    #> [1] "outfile.txt"
    dataParams(dh)
    #> [1] "input: Hello World!; outfile: outfile"
    dataNotes(dh)
    #> [1] "echo hello world txt"
    dataTags(dh)
    #> [1] ""
    dataYml(dh)
    #> [1] "/private/tmp/RtmpDamyzk/SharedData/echo_out_Hello_World!_outfile.yml"

`ReUseData`, as the name suggests, commits to promoting the data reuse. Data can be prepared in standard input formats (`toList`), e.g., YAML and JSON, to be easily integrated in workflow methods that are locally or cloud-hosted.

    (dh1 <- dataSearch(c("echo", "hello", "world")))
    #> dataHub with 1 records
    #> cache path:  /Users/biocbuild/Library/Caches/org.R-project.R/R/ReUseData
    #> # dataUpdate() to update the local data cache
    #> # dataSearch() to query a specific dataset
    #> # Additional information can be retrieved using:
    #> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
    #>
    #>              name        Path
    #>   BFC12535 | outfile.txt /tmp/RtmpDamyzk/SharedData/outfile.txt
    toList(dh1)
    #> $outfile.txt
    #> [1] "/tmp/RtmpDamyzk/SharedData/outfile.txt"
    toList(dh1, format = "yaml")
    #> [1] "outfile.txt: /tmp/RtmpDamyzk/SharedData/outfile.txt"
    toList(dh1, format = "json", file = file.path(tempdir(), "data.json"))
    #> File is saved as: "/tmp/RtmpDamyzk/data.json"
    #> {
    #>   "outfile.txt": "/tmp/RtmpDamyzk/SharedData/outfile.txt"
    #> }

Data can also be aggregated from different resources by tagging with specific software tools.

    dataSearch()
    #> dataHub with 1 records
    #> cache path:  /Users/biocbuild/Library/Caches/org.R-project.R/R/ReUseData
    #> # dataUpdate() to update the local data cache
    #> # dataSearch() to query a specific dataset
    #> # Additional information can be retrieved using:
    #> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
    #>
    #>              name        Path
    #>   BFC12535 | outfile.txt /tmp/RtmpDamyzk/SharedData/outfile.txt
    dataTags(dh[1]) <- "#gatk"
    dataSearch("#gatk")
    #> dataHub with 1 records
    #> cache path:  /Users/biocbuild/Library/Caches/org.R-project.R/R/ReUseData
    #> # dataUpdate() to update the local data cache
    #> # dataSearch() to query a specific dataset
    #> # Additional information can be retrieved using:
    #> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
    #>
    #>              name        Path
    #>   BFC12535 | outfile.txt /tmp/RtmpDamyzk/SharedData/outfile.txt

NOTE: if the argument `cloud=TRUE` is enabled, `dataUpdate()` will also cache the pregenerated data sets (from evaluation of public ReUseData recipes) that are available on ReUseData google bucket and return in the `dataHub` object that are fully searchable. Please see the following section for details.

# 3 Cloud data resources

With the prebuilt data recipes for curation (e.g., downloading, unzipping, indexing) of commonly used public data resources we have pregenerated some data sets and put them on the cloud space for direct use.

Before searching, one need to use `dataUpdate(cloud=TRUE)` to sync the existing data sets on cloud, then `dataSearch()` can be used to search any available data set either in local cache and on the cloud.

    gcpdir <- file.path(tempdir(), "gcpData")
    dataUpdate(gcpdir, cloud=TRUE)
    #>
    #> Updating data record...
    #> 14bf7e81b622_GRCh38.primary_assembly.genome.fa.1.bt2 added
    #> 14bf765dc799e_GRCh38.primary_assembly.genome.fa.2.bt2 added
    #> 14bf76db4a850_GRCh38.primary_assembly.genome.fa.3.bt2 added
    #> 14bf76f965474_GRCh38.primary_assembly.genome.fa.4.bt2 added
    #> 14bf77682c4e7_GRCh38.primary_assembly.genome.fa.rev.1.bt2 added
    #> 14bf734d567a_GRCh38.primary_assembly.genome.fa.rev.2.bt2 added
    #> 14bf74a686547_outfile.txt added
    #> 14bf7bd13e7b_GRCh37_to_GRCh38.chain added
    #> 14bf7565d034c_GRCh37_to_NCBI34.chain added
    #> 14bf774839edf_GRCh37_to_NCBI35.chain added
    #> 14bf76d337e3b_GRCh37_to_NCBI36.chain added
    #> 14bf74fa48b7f_GRCh38_to_GRCh37.chain added
    #> 14bf73bc267b2_GRCh38_to_NCBI34.chain added
    #> 14bf75825f5c4_GRCh38_to_NCBI35.chain added
    #> 14bf7242a4412_GRCh38_to_NCBI36.chain added
    #> 14bf752db0c4a_NCBI34_to_GRCh37.chain added
    #> 14bf72b03f8c5_NCBI34_to_GRCh38.chain added
    #> 14bf711c16393_NCBI35_to_GRCh37.chain added
    #> 14bf72f705100_NCBI35_to_GRCh38.chain added
    #> 14bf776d5ef54_NCBI36_to_GRCh37.chain added
    #> 14bf75753b0bf_NCBI36_to_GRCh38.chain added
    #> 14bf73778f863_GRCm38_to_NCBIM36.chain added
    #> 14bf766fb4808_GRCm38_to_NCBIM37.chain added
    #> 14bf77b363a09_NCBIM36_to_GRCm38.chain added
    #> 14bf7251c6411_NCBIM37_to_GRCm38.chain added
    #> 14bf76aedab1f_1000G_omni2.5.b37.vcf.gz added
    #> 14bf7197db711_1000G_omni2.5.b37.vcf.gz.tbi added
    #> 14bf7c7dca2a_Mills_and_1000G_gold_standard.indels.b37.vcf.gz added
    #> 14bf7166391ce_Mills_and_1000G_gold_standard.indels.b37.vcf.gz.tbi added
    #> 14bf762f976dd_1000G_omni2.5.hg38.vcf.gz added
    #> 14bf767ead9ee_1000G_omni2.5.hg38.vcf.gz.tbi added
    #> 14bf76b89cd8e_Mills_and_1000G_gold_standard.indels.hg38.vcf.gz added
    #> 14bf724165cca_Mills_and_1000G_gold_standard.indels.hg38.vcf.gz.tbi added
    #> 14bf73825e448_af-only-gnomad.raw.sites.vcf added
    #> 14bf73fae4fc4_af-only-gnomad.raw.sites.vcf.idx added
    #> 14bf74cf6ed85_Mutect2-exome-panel.vcf.idx added
    #> 14bf76763de3c_Mutect2-WGS-panel-b37.vcf added
    #> 14bf74d936a2b_Mutect2-WGS-panel-b37.vcf.idx added
    #> 14bf791f54d7_small_exac_common_3.vcf added
    #> 14bf767faf3ee_small_exac_common_3.vcf.idx added
    #> 14bf7ca4c397_1000g_pon.hg38.vcf.gz added
    #> 14bf71524f4fd_1000g_pon.hg38.vcf.gz.tbi added
    #> 14bf72d5018e3_af-only-gnomad.hg38.vcf.gz added
    #> 14bf76591f652_af-only-gnomad.hg38.vcf.gz.tbi added
    #> 14bf751c2b596_small_exac_common_3.hg38.vcf.gz added
    #> 14bf7461fb4c9_small_exac_common_3.hg38.vcf.gz.tbi added
    #> 14bf74b961c16_gencode.v41.annotation.gtf added
    #> 14bf76c0e0f1e_gencode.v42.annotation.gtf added
    #> 14bf7f02b1fe_gencode.vM30.annotation.gtf added
    #> 14bf779f3a264_gencode.vM31.annotation.gtf added
    #> 14bf76a2a91c8_gencode.v41.transcripts.fa added
    #> 14bf710c917ec_gencode.v41.transcripts.fa.fai added
    #> 14bf72418f90_gencode.v42.transcripts.fa added
    #> 14bf71e383818_gencode.v42.transcripts.fa.fai added
    #> 14bf77ceabf27_gencode.vM30.pc_transcripts.fa added
    #> 14bf717abd983_gencode.vM30.pc_transcripts.fa.fai added
    #> 14bf713553399_gencode.vM31.pc_transcripts.fa added
    #> 14bf73eae8bb9_gencode.vM31.pc_transcripts.fa.fai added
    #> 14bf735573ed5_GRCh38.primary_assembly.genome.fa.1.ht2 added
    #> 14bf772de2d4e_GRCh38.primary_assembly.genome.fa.2.ht2 added
    #> 14bf7587096cc_GRCh38.primary_assembly.genome.fa.3.ht2 added
    #> 14bf747bc5870_GRCh38.primary_assembly.genome.fa.4.ht2 added
    #> 14bf71e5245db_GRCh38.primary_assembly.genome.fa.5.ht2 added
    #> 14bf72b683c6a_GRCh38.primary_assembly.genome.fa.6.ht2 added
    #> 14bf748566969_GRCh38.primary_assembly.genome.fa.7.ht2 added
    #> 14bf721229199_GRCh38.primary_assembly.genome.fa.8.ht2 added
    #> 14bf76484e4cd_GRCh38_full_analysis_set_plus_decoy_hla.fa.fai added
    #> 14bf750c98249_GRCh38_full_analysis_set_plus_decoy_hla.fa.amb added
    #> 14bf75d88b00e_GRCh38_full_analysis_set_plus_decoy_hla.fa.ann added
    #> 14bf738de971b_GRCh38_full_analysis_set_plus_decoy_hla.fa.bwt added
    #> 14bf71d928ac8_GRCh38_full_analysis_set_plus_decoy_hla.fa.pac added
    #> 14bf77fd55fa2_GRCh38_full_analysis_set_plus_decoy_hla.fa.sa added
    #> 14bf71179c63f_GRCh38_full_analysis_set_plus_decoy_hla.fa added
    #> 14bf751c65b0f_GRCh38.primary_assembly.genome.fa.fai added
    #> 14bf735845fba_GRCh38.primary_assembly.genome.fa.amb added
    #> 14bf75a8c7c9_GRCh38.primary_assembly.genome.fa.ann added
    #> 14bf7bd46006_GRCh38.primary_assembly.genome.fa.bwt added
    #> 14bf723ec2ffb_GRCh38.primary_assembly.genome.fa.pac added
    #> 14bf767421a29_GRCh38.primary_assembly.genome.fa.sa added
    #> 14bf724c3aeb5_GRCh38.primary_assembly.genome.fa added
    #> 14bf72b02ffee_hs37d5.fa.fai added
    #> 14bf751f07851_hs37d5.fa.amb added
    #> 14bf726b37de_hs37d5.fa.ann added
    #> 14bf74d20d10f_hs37d5.fa.bwt added
    #> 14bf7257d5758_hs37d5.fa.pac added
    #> 14bf747f165a2_hs37d5.fa.sa added
    #> 14bf739479394_hs37d5.fa added
    #> 14bf7a29f6ed_complete_ref_lens.bin added
    #> 14bf7491246d1_ctable.bin added
    #> 14bf74ee768d1_ctg_offsets.bin added
    #> 14bf73a9299cf_duplicate_clusters.tsv added
    #> 14bf76eb00313_info.json added
    #> 14bf7659a0d2a_mphf.bin added
    #> 14bf764d67482_pos.bin added
    #> 14bf73b7b3686_pre_indexing.log added
    #> 14bf71638b5ec_rank.bin added
    #> 14bf7652fa459_ref_indexing.log added
    #> 14bf71acf02f5_refAccumLengths.bin added
    #> 14bf7ccb3093_reflengths.bin added
    #> 14bf76fe20974_refseq.bin added
    #> 14bf760dad80e_seq.bin added
    #> 14bf73f9eb0cf_versionInfo.json added
    #> 14bf74b6a06aa_salmon_index added
    #> 14bf71ddba994_chrLength.txt added
    #> 14bf7405a42dc_chrName.txt added
    #> 14bf765db9a85_chrNameLength.txt added
    #> 14bf7347dc601_chrStart.txt added
    #> 14bf72d528693_exonGeTrInfo.tab added
    #> 14bf75013424_exonInfo.tab added
    #> 14bf712062a0d_geneInfo.tab added
    #> 14bf752b2c4b9_Genome added
    #> 14bf756917819_genomeParameters.txt added
    #> 14bf76863ddb5_Log.out added
    #> 14bf77470cb9d_SA added
    #> 14bf71947ec24_SAindex added
    #> 14bf740e03c73_sjdbInfo.txt added
    #> 14bf741a0c34b_sjdbList.fromGTF.out.tab added
    #> 14bf721759296_sjdbList.out.tab added
    #> 14bf72deac703_transcriptInfo.tab added
    #> 14bf710afad82_GRCh38.GENCODE.v42_100 added
    #> 14bf77da83a5c_knownGene_hg38.sql added
    #> 14bf72f8fae77_knownGene_hg38.txt added
    #> 14bf72071f06_refGene_hg38.sql added
    #> 14bf72185c3f4_refGene_hg38.txt added
    #> 14bf75503d95d_knownGene_mm39.sql added
    #> 14bf76fb39445_knownGene_mm39.txt added
    #> 14bf776cb774d_refGene_mm39.sql added
    #> 14bf728059d29_refGene_mm39.txt added
    #> dataHub with 128 records
    #> cache path:  /Users/biocbuild/Library/Caches/org.R-project.R/R/ReUseData
    #> # dataUpdate() to update the local data cache
    #> # dataSearch() to query a specific dataset
    #> # Additional information can be retrieved using:
    #> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
    #>
    #>              name
    #>   BFC12536 | GRCh38.primary_assembly.genome.fa.1.bt2
    #>   BFC12537 | GRCh38.primary_assembly.genome.fa.2.bt2
    #>   BFC12538 | GRCh38.primary_assembly.genome.fa.3.bt2
    #>   BFC12539 | GRCh38.primary_assembly.genome.fa.4.bt2
    #>   BFC12540 | GRCh38.primary_assembly.genome.fa.rev.1.bt2
    #>   ...        ...
    #>   BFC12659 | refGene_hg38.txt
    #>   BFC12660 | knownGene_mm39.sql
    #>   BFC12661 | knownGene_mm39.txt
    #>   BFC12662 | refGene_mm39.sql
    #>   BFC12663 | refGene_mm39.txt
    #>            Path
    #>   BFC12536 https://storage.googleapis.com/reusedata/bowtie2_index/GRCh38.pr...
    #>   BFC12537 https://storage.googleapis.com/reusedata/bowtie2_index/GRCh38.pr...
    #>   BFC12538 https://storage.googleapis.com/reusedata/bowtie2_index/GRCh38.pr...
    #>   BFC12539 https://storage.googleapis.com/reusedata/bowtie2_index/GRCh38.pr...
    #>   BFC12540 https://storage.googleapis.com/reusedata/bowtie2_index/GRCh38.pr...
    #>   ...      ...
    #>   BFC12659 https://storage.googleapis.com/reusedata/ucsc_database/refGene_h...
    #>   BFC12660 https://storage.googleapis.com/reusedata/ucsc_database/knownGene...
    #>   BFC12661 https://storage.googleapis.com/reusedata/ucsc_database/knownGene...
    #>   BFC12662 https://storage.googleapis.com/reusedata/ucsc_database/refGene_m...
    #>   BFC12663 https://storage.googleapis.com/reusedata/ucsc_database/refGene_m...

If the data of interest already exist on the cloud, then `getCloudData` will directly download the data to your computer. Add it to the local caching system using `dataUpdate()` for later use.

    (dh <- dataSearch(c("ensembl", "GRCh38")))
    #> dataHub with 8 records
    #> cache path:  /Users/biocbuild/Library/Caches/org.R-project.R/R/ReUseData
    #> # dataUpdate() to update the local data cache
    #> # dataSearch() to query a specific dataset
    #> # Additional information can be retrieved using:
    #> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
    #>
    #>              name
    #>   BFC12543 | GRCh37_to_GRCh38.chain
    #>   BFC12547 | GRCh38_to_GRCh37.chain
    #>   BFC12548 | GRCh38_to_NCBI34.chain
    #>   BFC12549 | GRCh38_to_NCBI35.chain
    #>   BFC12550 | GRCh38_to_NCBI36.chain
    #>   BFC12552 | NCBI34_to_GRCh38.chain
    #>   BFC12554 | NCBI35_to_GRCh38.chain
    #>   BFC12556 | NCBI36_to_GRCh38.chain
    #>            Path
    #>   BFC12543 https://storage.googleapis.com/reusedata/ensembl_liftover/GRCh37...
    #>   BFC12547 https://storage.googleapis.com/reusedata/ensembl_liftover/GRCh38...
    #>   BFC12548 https://storage.googleapis.com/reusedata/ensembl_liftover/GRCh38...
    #>   BFC12549 https://storage.googleapis.com/reusedata/ensembl_liftover/GRCh38...
    #>   BFC12550 https://storage.googleapis.com/reusedata/ensembl_liftover/GRCh38...
    #>   BFC12552 https://storage.googleapis.com/reusedata/ensembl_liftover/NCBI34...
    #>   BFC12554 https://storage.googleapis.com/reusedata/ensembl_liftover/NCBI35...
    #>   BFC12556 https://storage.googleapis.com/reusedata/ensembl_liftover/NCBI36...
    getCloudData(dh[1], outdir = gcpdir)
    #> Data is downloaded:
    #> /tmp/RtmpDamyzk/gcpData/GRCh37_to_GRCh38.chain

Now we create the data cache with only local data files, and we can see that the downloaded data is available.

    dataUpdate(gcpdir)  ## Update local data cache (without cloud data)
    #>
    #> Updating data record...
    #> GRCh37_to_GRCh38.chain added
    #> Warning in dm$tag[idx] <- value: number of items to replace is not a multiple
    #> of replacement length
    #> dataHub with 1 records
    #> cache path:  /Users/biocbuild/Library/Caches/org.R-project.R/R/ReUseData
    #> # dataUpdate() to update the local data cache
    #> # dataSearch() to query a specific dataset
    #> # Additional information can be retrieved using:
    #> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
    #>
    #>              name
    #>   BFC12664 | GRCh37_to_GRCh38.chain
    #>            Path
    #>   BFC12664 /tmp/RtmpDamyzk/gcpData/GRCh37_to_GRCh38.chain
    dataSearch()  ## data is available locally!!!
    #> dataHub with 1 records
    #> cache path:  /Users/biocbuild/Library/Caches/org.R-project.R/R/ReUseData
    #> # dataUpdate() to update the local data cache
    #> # dataSearch() to query a specific dataset
    #> # Additional information can be retrieved using:
    #> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
    #>
    #>              name
    #>   BFC12664 | GRCh37_to_GRCh38.chain
    #>            Path
    #>   BFC12664 /tmp/RtmpDamyzk/gcpData/GRCh37_to_GRCh38.chain

The data supports user-friendly discovery and access through the `ReUseData` portal, where detailed instructions are provided for straight-forward incorporation into data analysis pipelines run on local computing nodes, web resources, and cloud computing platforms (e.g., Terra, CGC).

# 4 Know your data

Here we provide a function `meta_data()` to create a data frame that contains all information about the data sets in the specified file path (recursively), including the annotation file (`$yml` column), parameter values for the recipe (`$params` column), data file path (`$output` column), keywords for data file (`notes` columns), date of data generation (`date` column), and any tag if available (`tag` column).

Use `cleanup = TRUE` to cleanup any invalid or expired/older intermediate files.

    mt <- meta_data(outdir)
    head(mt)
    #>                                                                    yml
    #> 1 /private/tmp/RtmpDamyzk/SharedData/echo_out_Hello_World!_outfile.yml
    #>                                  params                                 output
    #> 1 input: Hello World!; outfile: outfile /tmp/RtmpDamyzk/SharedData/outfile.txt
    #>                  notes                      date
    #> 1 echo hello world txt 2023-07-31 14:47:57.88844

# 5 SessionInfo

    sessionInfo()
    #> R version 4.3.1 (2023-06-16)
    #> Platform: aarch64-apple-darwin20 (64-bit)
    #> Running under: macOS Monterey 12.6.7
    #>
    #> Matrix products: default
    #> BLAS:   /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRblas.0.dylib
    #> LAPACK: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.11.0
    #>
    #> locale:
    #> [1] C/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
    #>
    #> time zone: America/New_York
    #> tzcode source: internal
    #>
    #> attached base packages:
    #> [1] stats4    stats     graphics  grDevices utils     datasets  methods
    #> [8] base
    #>
    #> other attached packages:
    #> [1] ReUseData_1.0.1     Rcwl_1.16.0         S4Vectors_0.38.1
    #> [4] BiocGenerics_0.46.0 yaml_2.3.7          BiocStyle_2.28.0
    #>
    #> loaded via a namespace (and not attached):
    #>  [1] tidyselect_1.2.0      dplyr_1.1.2           blob_1.2.4
    #>  [4] filelock_1.0.2        R.utils_2.12.2        fastmap_1.1.1
    #>  [7] BiocFileCache_2.8.0   promises_1.2.0.1      digest_0.6.31
    #> [10] base64url_1.4         mime_0.12             lifecycle_1.0.3
    #> [13] ellipsis_0.3.2        RSQLite_2.3.1         magrittr_2.0.3
    #> [16] compiler_4.3.1        rlang_1.1.1           sass_0.4.6
    #> [19] progress_1.2.2        tools_4.3.1           utf8_1.2.3
    #> [22] data.table_1.14.8     knitr_1.43            prettyunits_1.1.1
    #> [25] brew_1.0-8            htmlwidgets_1.6.2     bit_4.0.5
    #> [28] curl_5.0.1            here_1.0.1            reticulate_1.30
    #> [31] RColorBrewer_1.1-3    batchtools_0.9.17     BiocParallel_1.34.2
    #> [34] purrr_1.0.1           withr_2.5.0           R.oo_1.25.0
    #> [37] grid_4.3.1            fansi_1.0.4           git2r_0.32.0
    #> [40] xtable_1.8-4          debugme_1.1.0         cli_3.6.1
    #> [43] rmarkdown_2.22        DiagrammeR_1.0.10     crayon_1.5.2
    #> [46] generics_0.1.3        httr_1.4.6            visNetwork_2.1.2
    #> [49] DBI_1.1.3             cachem_1.0.8          parallel_4.3.1
    #> [52] BiocManager_1.30.21   basilisk_1.12.1       vctrs_0.6.3
    #> [55] Matrix_1.5-4.1        jsonlite_1.8.5        dir.expiry_1.8.0
    #> [58] bookdown_0.34         hms_1.1.3             bit64_4.0.5
    #> [61] jquerylib_0.1.4       RcwlPipelines_1.16.0  glue_1.6.2
    #> [64] codetools_0.2-19      stringi_1.7.12        later_1.3.1
    #> [67] tibble_3.2.1          pillar_1.9.0          basilisk.utils_1.12.1
    #> [70] rappdirs_0.3.3        htmltools_0.5.5       R6_2.5.1
    #> [73] dbplyr_2.3.2          rprojroot_2.0.3       evaluate_0.21
    #> [76] shiny_1.7.4           lattice_0.21-8        R.methodsS3_1.8.2
    #> [79] png_0.1-8             backports_1.4.1       memoise_2.0.1
    #> [82] httpuv_1.6.11         bslib_0.5.0           Rcpp_1.0.10
    #> [85] checkmate_2.2.0       xfun_0.39             pkgconfig_2.0.3