# VCFArray: DelayedArray objects with on-disk/remote VCF backend

Qian Liu1 and Martin Morgan1

1Roswell Park Comprehensive Cancer Center, Buffalo, NY

#### last edit: 08/23/2018

#### Package

VCFArray 1.6.0

# 1 Introduction

VCFArray is a Bioconductor package that represents VCF files as objects derived from the DelayedArray package and `DelayedArray` class. It converts data entries from VCF file into a `DelayedArray`-derived data structure. The backend VCF file could either be saved on-disk locally or remote as online resources. Data entries that could be extracted include the fixed data fields (REF, ALT, QUAL, FILTER), information field (e.g., AA, AF…), and the individual format field (e.g., GT, DP…). The array data generated from fixed/information fields are one-dimensional `VCFArray`, with the dimension being the length of the variants. The array data generated from individual `FORMAT` field are always returned with the first dimension being `variants` and the second dimension being `samples`. This feature is consistent with the assay data saved in `SummarizedExperiment`, and makes the `VCFArray` package interoperable with other established Bioconductor data infrastructure.

# 2 Installation

1. Download the package.

    if (!requireNamespace("BiocManager", quietly = TRUE))
        install.packages("BiocManager")
    BiocManager::install("VCFArray")

The development version is also available to download from Github.

    BiocManager::install("Bioconductor/VCFArray")

2. Load the package into R session.

    library(VCFArray)

# 3 VCFArray

## 3.1 VCFArray constructor

To construct a `VCFArray` object, 4 arguments are needed: `file`, `vindex` and `name`, and `pfix`. The `file` argument could take either a character string (VCF file name), or `VcfFile` object, or a `RangedVcfStack` object. `name` argument must be specified to indicate which data entry we want to extract from the input file. It’s case-sensitive, and must be consistent with the names from VCF header file. `vindex` argument will only be used to indicate the file path of the index file if it does not exist. `pfix` is used to spefify the category that the `name` field belongs to. NOTE that the `pfix` needs to be provided specifically when there are same `name` in multiple categories, otherwise, error will return.

The `vcfFields()` method takes the VCF file path, `VcfFile` object or `RangedVcfStack` object as input, and returns a CharacterList with all available VCF fields within specific categories. Users should consult the `fixed`, `info` and `geno` category for available data entries that could be converted into `VCFArray` instances. The data entry names can be used as input for the `name` argument in `VCFArray` constructor.

    args(VCFArray)
    #> function (file, vindex = character(), name = NA, pfix = NULL)
    #> NULL
    fl <- system.file("extdata", "chr22.vcf.gz", package = "VariantAnnotation")
    library(VariantAnnotation)
    vcfFields(fl)
    #> CharacterList of length 4
    #> [["fixed"]] REF ALT QUAL FILTER
    #> [["info"]] LDAF AVGPOST RSQ ERATE THETA ... ASN_AF AFR_AF EUR_AF VT SNPSOURCE
    #> [["geno"]] GT DS GL
    #> [["samples"]] HG00096 HG00097 HG00099 HG00100 HG00101

Since the index file for our vcf file already exists, the `vindex` argument would not be needed (which is the most common case for on-disk VCF files). So we can construct the `VCFArray` object for the `GT` data entry in the provided VCF file with arguments of `file` and `name` only.

    VCFArray(file = fl, name = "GT")
    #> <10376 x 5> matrix of class VCFMatrix and type "character":
    #>             HG00096 HG00097 HG00099 HG00100 HG00101
    #> rs7410291   "0|0"   "0|0"   "1|0"   "0|0"   "0|0"
    #> rs147922003 "0|0"   "0|0"   "0|0"   "0|0"   "0|0"
    #> rs114143073 "0|0"   "0|0"   "0|0"   "0|0"   "0|0"
    #> ...         .       .       .       .       .
    #> rs5770892   "0|0"   "0|0"   "0|0"   "0|0"   "0|1"
    #> rs144055359 "0|0"   "0|0"   "0|0"   "0|0"   "0|0"
    #> rs114526001 "0|0"   "0|0"   "0|0"   "0|0"   "0|0"

We can also construct a `VCFArray` object with the `file` argument being a `VcfFile` object.

    vcf <- VariantAnnotation::VcfFile(fl)
    VCFArray(file = vcf, name = "DS")
    #> <10376 x 5> matrix of class VCFMatrix and type "double":
    #>             HG00096 HG00097 HG00099 HG00100 HG00101
    #>   rs7410291       0       0       1       0       0
    #> rs147922003       0       0       0       0       0
    #> rs114143073       0       0       0       0       0
    #>         ...       .       .       .       .       .
    #>   rs5770892       0       0       0       0       1
    #> rs144055359       0       0       0       0       0
    #> rs114526001       0       0       0       0       0

The `file` argument could also take `RangedVcfStack` object. Note that an ordinary `VcfStack` object without `Range` information could not be used to construct a `VCFArray`.

    extdata <- system.file(package = "GenomicFiles", "extdata")
    files <- dir(extdata, pattern="^CEUtrio.*bgz$", full=TRUE)[1:2]
    names(files) <- sub(".*_([0-9XY]+).*", "\\1", basename(files))
    seqinfo <- as(readRDS(file.path(extdata, "seqinfo.rds")), "Seqinfo")
    stack <- GenomicFiles::VcfStack(files, seqinfo)
    gr <- as(GenomicFiles::seqinfo(stack)[rownames(stack)], "GRanges")
    ## RangedVcfStack
    rgstack <- GenomicFiles::RangedVcfStack(stack, rowRanges = gr)
    rgstack
    #> VcfStack object with 2 files and 3 samples
    #> GRanges object with 2 ranges and 0 metadata columns
    #> Seqinfo object with 25 sequences from hg19 genome
    #> use 'readVcfStack()' to extract VariantAnnotation VCF.

Here we choose the `name = SB`, which returns a 3-dimensional `VCFArray` object, with the first 2 dimensions correspond to variants and samples respectively.

    vcfFields(rgstack)$geno
    #> [1] "GT"     "AD"     "DP"     "GQ"     "MIN_DP" "PGT"    "PID"    "PL"
    #> [9] "SB"
    VCFArray(rgstack, name = "SB")
    #> <318 x 3 x 4> array of class VCFArray and type "integer":
    #> ,,1
    #>             NA12878 NA12891 NA12892
    #> rs771898125      NA      NA      NA
    #>   rs6118055      NA      NA      NA
    #>         ...       .       .       .
    #>   rs2009307      NA      NA      NA
    #>   rs2273444      NA      NA      NA
    #>
    #> ...
    #>
    #> ,,4
    #>             NA12878 NA12891 NA12892
    #> rs771898125      NA      NA      NA
    #>   rs6118055      NA      NA      NA
    #>         ...       .       .       .
    #>   rs2009307      NA      NA      NA
    #>   rs2273444      NA      NA      NA

As the vignette title suggest, the backend VCF file could also be remote files. Here we included an example of representing VCF file of chromosome 22 from the 1000 Genomes Project (Phase 3). NOTE that for a remote VCF file, the `vindex` argument must be specified. Since this VCF files is relatively big, and it takes longer time, we only show the code here without evaluation.

    chr22url <- "ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20130502/ALL.chr22.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz"
    chr22url.tbi <- paste0(chr22url, ".tbi")
    va <- VCFArray(chr22url, vindex =chr22url.tbi, name = "GT")

## 3.2 VCFArray methods

`VCFArray` represents VCF files as `DelayedArray` instances. It has methods like `dim`, `dimnames` defined, and it inherits array-like operations and methods from `DelayedArray`, e.g., the subsetting method of `[`.
NOTE that for 1-dimensional `VCFArray` objects that are generated from the fixed / information data field of VCF file, `drop = FALSE` should always be used with `[` subsetting to ensure `VCFArray` object as returned value.

### 3.2.1 slot accessors

`seed` returns the `VCFArraySeed` of the `VCFArray` object, which includes information about the backend VCF file, e.g., the vcf file path, index file path, name of the data entry (with a prefix of category), dimension and etc.

    va <- VCFArray(fl, name = "GT")
    seed(va)
    #> VCFArraySeed
    #> VCF file path: C:/Users/biocbuild/bbs-3.12-bioc/R/library/VariantAnnotation/extdata/chr22.vcf.gz
    #> VCF index path: C:/Users/biocbuild/bbs-3.12-bioc/R/library/VariantAnnotation/extdata/chr22.vcf.gz.tbi
    #> array data: GT
    #> dim: 10376 x 5

`vcffile` returns the `VcfFile` object corresponding to the backend VCF file.

    vcffile(va)
    #> class: VcfFile
    #> path: C:/Users/biocbuild/bbs-3.12-bioc/R/library/VariantAnnotati.../chr22.vcf.gz
    #> index: C:/Users/biocbuild/bbs-3.12-bioc/R/library/VariantAnn.../chr22.vcf.gz.tbi
    #> isOpen: FALSE
    #> yieldSize: NA

### 3.2.2 `dim()` and `dimnames()`

The `dimnames(VCFArray)` returns an unnamed list, with the length of each element being the same as return from `dim(VCFArray)`.

    va <- VCFArray(fl, name = "GT")
    dim(va)
    #> [1] 10376     5
    class(dimnames(va))
    #> [1] "list"
    lengths(dimnames(va))
    #> [1] 10376     5

### 3.2.3 `[` subsetting

`VCFArray` instances can be subsetted, following the usual R conventions, with numeric or logical vectors; logical vectors are recycled to the appropriate length.

    va[1:3, 1:3]
    #> <3 x 3> matrix of class DelayedMatrix and type "character":
    #>             HG00096 HG00097 HG00099
    #> rs7410291   "0|0"   "0|0"   "1|0"
    #> rs147922003 "0|0"   "0|0"   "0|0"
    #> rs114143073 "0|0"   "0|0"   "0|0"
    va[c(TRUE, FALSE), ]
    #> <5188 x 5> matrix of class DelayedMatrix and type "character":
    #>             HG00096 HG00097 HG00099 HG00100 HG00101
    #> rs7410291   "0|0"   "0|0"   "1|0"   "0|0"   "0|0"
    #> rs114143073 "0|0"   "0|0"   "0|0"   "0|0"   "0|0"
    #> rs182170314 "0|0"   "0|0"   "0|0"   "0|0"   "0|0"
    #> ...         .       .       .       .       .
    #> rs9628212   "0|0"   "0|0"   "0|0"   "0|0"   "0|0"
    #> rs9628178   "0|0"   "0|0"   "0|0"   "0|0"   "0|0"
    #> rs144055359 "0|0"   "0|0"   "0|0"   "0|0"   "0|0"

### 3.2.4 Some numeric calculation

Numeric calculations could be evaluated on `VCFArray` objects.

    ds <- VCFArray(fl, name = "DS")
    log(ds+5)
    #> <10376 x 5> matrix of class DelayedMatrix and type "double":
    #>              HG00096  HG00097  HG00099  HG00100  HG00101
    #>   rs7410291 1.609438 1.609438 1.791759 1.609438 1.609438
    #> rs147922003 1.609438 1.609438 1.609438 1.609438 1.609438
    #> rs114143073 1.609438 1.609438 1.609438 1.609438 1.609438
    #>         ...        .        .        .        .        .
    #>   rs5770892 1.609438 1.609438 1.609438 1.609438 1.791759
    #> rs144055359 1.609438 1.609438 1.609438 1.609438 1.609438
    #> rs114526001 1.609438 1.609438 1.609438 1.609438 1.609438

# 4 Internals: VCFArraySeed

The `VCFArraySeed` class represents the ‘seed’ for the `VCFArray` object. It is not exported from the VCFArray package. Seed objects should contain the VCF file path, and are expected to satisfy the “seed contract” of DelayedArray, i.e. to support `dim()` and `dimnames()`.

    seed <- VCFArray:::VCFArraySeed(fl, name = "GT", pfix = NULL)
    seed
    #> VCFArraySeed
    #> VCF file path: C:/Users/biocbuild/bbs-3.12-bioc/R/library/VariantAnnotation/extdata/chr22.vcf.gz
    #> VCF index path: C:/Users/biocbuild/bbs-3.12-bioc/R/library/VariantAnnotation/extdata/chr22.vcf.gz.tbi
    #> array data: GT
    #> dim: 10376 x 5
    path(vcffile(seed))
    #> [1] "C:/Users/biocbuild/bbs-3.12-bioc/R/library/VariantAnnotation/extdata/chr22.vcf.gz"

The seed can be used to construct a `VCFArray` instance.

    (va <- VCFArray(seed))
    #> <10376 x 5> matrix of class VCFMatrix and type "character":
    #>             HG00096 HG00097 HG00099 HG00100 HG00101
    #> rs7410291   "0|0"   "0|0"   "1|0"   "0|0"   "0|0"
    #> rs147922003 "0|0"   "0|0"   "0|0"   "0|0"   "0|0"
    #> rs114143073 "0|0"   "0|0"   "0|0"   "0|0"   "0|0"
    #> ...         .       .       .       .       .
    #> rs5770892   "0|0"   "0|0"   "0|0"   "0|0"   "0|1"
    #> rs144055359 "0|0"   "0|0"   "0|0"   "0|0"   "0|0"
    #> rs114526001 "0|0"   "0|0"   "0|0"   "0|0"   "0|0"

The `DelayedArray()` constructor with `VCFArraySeed` object as inputs will return the same content as the `VCFArray()` constructor over the same `VCFArraySeed`.

    da <- DelayedArray(seed)
    class(da)
    #> [1] "VCFMatrix"
    #> attr(,"package")
    #> [1] "VCFArray"
    all.equal(da, va)
    #> [1] TRUE

# 5 sessionInfo()

    sessionInfo()
    #> R version 4.0.3 (2020-10-10)
    #> Platform: x86_64-w64-mingw32/x64 (64-bit)
    #> Running under: Windows Server 2012 R2 x64 (build 9600)
    #>
    #> Matrix products: default
    #>
    #> locale:
    #> [1] LC_COLLATE=C
    #> [2] LC_CTYPE=English_United States.1252
    #> [3] LC_MONETARY=English_United States.1252
    #> [4] LC_NUMERIC=C
    #> [5] LC_TIME=English_United States.1252
    #>
    #> attached base packages:
    #> [1] stats4    parallel  stats     graphics  grDevices utils     datasets
    #> [8] methods   base
    #>
    #> other attached packages:
    #>  [1] VariantAnnotation_1.36.0    Rsamtools_2.6.0
    #>  [3] Biostrings_2.58.0           XVector_0.30.0
    #>  [5] SummarizedExperiment_1.20.0 Biobase_2.50.0
    #>  [7] GenomicRanges_1.42.0        GenomeInfoDb_1.26.0
    #>  [9] VCFArray_1.6.0              DelayedArray_0.16.0
    #> [11] IRanges_2.24.0              S4Vectors_0.28.0
    #> [13] MatrixGenerics_1.2.0        matrixStats_0.57.0
    #> [15] Matrix_1.2-18               BiocGenerics_0.36.0
    #> [17] BiocStyle_2.18.0
    #>
    #> loaded via a namespace (and not attached):
    #>  [1] httr_1.4.2               bit64_4.0.5              GenomicFiles_1.26.0
    #>  [4] assertthat_0.2.1         askpass_1.1              BiocManager_1.30.10
    #>  [7] BiocFileCache_1.14.0     blob_1.2.1               BSgenome_1.58.0
    #> [10] GenomeInfoDbData_1.2.4   yaml_2.2.1               progress_1.2.2
    #> [13] pillar_1.4.6             RSQLite_2.2.1            lattice_0.20-41
    #> [16] glue_1.4.2               digest_0.6.27            htmltools_0.5.0
    #> [19] XML_3.99-0.5             pkgconfig_2.0.3          biomaRt_2.46.0
    #> [22] bookdown_0.21            zlibbioc_1.36.0          purrr_0.3.4
    #> [25] BiocParallel_1.24.0      tibble_3.0.4             openssl_1.4.3
    #> [28] generics_0.0.2           ellipsis_0.3.1           GenomicFeatures_1.42.0
    #> [31] magrittr_1.5             crayon_1.3.4             memoise_1.1.0
    #> [34] evaluate_0.14            xml2_1.3.2               tools_4.0.3
    #> [37] prettyunits_1.1.1        hms_0.5.3                lifecycle_0.2.0
    #> [40] stringr_1.4.0            AnnotationDbi_1.52.0     compiler_4.0.3
    #> [43] rlang_0.4.8              grid_4.0.3               RCurl_1.98-1.2
    #> [46] rappdirs_0.3.1           bitops_1.0-6             rmarkdown_2.5
    #> [49] DBI_1.1.0                curl_4.3                 R6_2.5.0
    #> [52] GenomicAlignments_1.26.0 knitr_1.30               dplyr_1.0.2
    #> [55] rtracklayer_1.50.0       bit_4.0.4                stringi_1.5.3
    #> [58] Rcpp_1.0.5               vctrs_0.3.4              dbplyr_1.4.4
    #> [61] tidyselect_1.1.0         xfun_0.18