AutoGrid {DelayedArray} | R Documentation

## Create automatic grids to use for block processing of array-like objects

### Description

We provide various utility functions to create grids that can be used for block processing of array-like objects:

- `defaultAutoGrid()` is the default automatic grid maker. It creates a grid that is suitable for block processing of the array-like object passed to it.

- `rowAutoGrid()` and `colAutoGrid()` are more specialized automatic grid makers, for the 2-dimensional case. They can be used to create a grid where the blocks are made of full rows or full columns, respectively.

- `defaultSinkAutoGrid()` is a specialized version of `defaultAutoGrid()` for creating a grid that is suitable for writing to a RealizationSink derivative while walking on it.

### Usage

    defaultAutoGrid(x, block.length=NULL, chunk.grid=NULL, block.shape=NULL)

    ## Two specialized "automatic grid makers" for the 2-dimensional case:
    rowAutoGrid(x, nrow=NULL, block.length=NULL)
    colAutoGrid(x, ncol=NULL, block.length=NULL)

    ## Replace default automatic grid maker with user-defined one:
    getAutoGridMaker()
    setAutoGridMaker(GRIDMAKER="defaultAutoGrid")

    ## A specialized version of defaultAutoGrid() to create an automatic
    ## grid on a RealizationSink derivative:
    defaultSinkAutoGrid(sink, block.length=NULL, chunk.grid=NULL)

### Arguments

`x` | An array-like or matrix-like object for `defaultAutoGrid`. A matrix-like object for `rowAutoGrid` and `colAutoGrid`.
`block.length` | The length of the blocks i.e. the number of array elements per block. By default the automatic block length (returned by `getAutoBlockLength(type(x))`, or `getAutoBlockLength(type(sink))` in the case of `defaultSinkAutoGrid()`) is used. Depending on how much memory is available on your machine, you might want to increase (or decrease) the automatic block length by adjusting the automatic block size with `setAutoBlockSize()`.
`chunk.grid` | The grid of physical chunks. By default `chunkGrid(x)` (or `chunkGrid(sink)` in the case of `defaultSinkAutoGrid()`) is used.
`block.shape` | A string specifying the shape of the blocks. See `makeCappedVolumeBox` for a description of the supported shapes. By default `getAutoBlockShape()` is used.
`nrow` | The number of rows of the blocks. The bottommost blocks might have less. See examples below.
`ncol` | The number of columns of the blocks. The rightmost blocks might have less. See examples below.
`GRIDMAKER` | The function to use as automatic grid maker, that is, the function that will be used by `blockApply()` and `blockReduce()` to make a grid when no grid is supplied via their `grid` argument. The function will be called on array-like object `x` and must return an ArrayGrid object, say `grid`, that is compatible with `x` i.e. such that `refdim(grid)` is identical to `dim(x)`. `GRIDMAKER` can be specified as a function or as a single string naming a function. It can be a user-defined function or a pre-defined grid maker like `defaultAutoGrid`, `rowAutoGrid`, or `colAutoGrid`. The automatic grid maker is set to `defaultAutoGrid` at package startup and can be reset anytime to this value by calling `setAutoGridMaker()` with no argument.
`sink` | A RealizationSink derivative.

### Details

By default, primary block processing functions `blockApply()` and `blockReduce()` use the grid returned by `defaultAutoGrid(x)` to walk on the blocks of array-like object `x`. This can be changed with `setAutoGridMaker()`.

By default `sinkApply()` uses the grid returned by `defaultSinkAutoGrid(sink)` to walk on the viewports of RealizationSink derivative `sink` and write to them.

### Value

`defaultAutoGrid`: An ArrayGrid object on reference array `x`. The grid elements define the blocks that will be used to process `x` by block. The grid is optimal in the sense that:

1. It's compatible with the grid of physical chunks a.k.a. chunk grid. This means that, when the chunk grid is known (i.e. when `chunkGrid(x)` is not NULL or `chunk.grid` is supplied), every block in the grid contains one or more full chunks. In other words, chunks never cross block boundaries.

2. Its resolution is such that the blocks have a length that is as close as possibe to (but does not exceed) `block.length`. An exception is made when some chunks already have a length that is >= `block.length`, in which case the returned grid is the same as the chunk grid.

Note that the returned grid is regular (i.e. is a RegularArrayGrid object) unless the chunk grid is not regular (i.e. is an ArbitraryArrayGrid object).

`rowAutoGrid`: A RegularArrayGrid object on reference array `x` where the grid elements define blocks made of full rows of `x`.

`colAutoGrid`: A RegularArrayGrid object on reference array `x` where the grid elements define blocks made of full columns of `x`.

`defaultSinkAutoGrid`: Like `defaultAutoGrid` except that `defaultSinkAutoGrid` always returns a grid with a "first-dim-grows-first" shape (note that, unlike the former, the latter has no `block.shape` argument). The advantage of using a grid with a "first-dim-grows-first" shape in the context of writing to the viewports of a RealizationSink derivative is that such a grid is guaranteed to work with "linear write only" realization backends. See important notes about "Cross realization backend compatibility" in `?write_block` for more information.

### See Also

- `setAutoBlockSize` and `setAutoBlockShape` to control the geometry of automatic blocks.

- `blockApply` and family for convenient block processing of an array-like object.

- ArrayGrid for the formal representation of grids and viewports.

- The `makeCappedVolumeBox` utility to make capped volume boxes.

- `chunkGrid`.

- `read_block` and `write_block`.

### Examples

    ## ---------------------------------------------------------------------
    ## A VERSION OF sum() THAT USES BLOCK PROCESSING
    ## ---------------------------------------------------------------------

    block_sum <- function(a, grid) {
      sums <- lapply(grid, function(viewport) sum(read_block(a, viewport)))
      sum(unlist(sums))
    }

    ## On an ordinary matrix:
    m <- matrix(runif(600), ncol=12)
    m_grid <- defaultAutoGrid(m, block.length=120)
    sum1 <- block_sum(m, m_grid)
    sum1

    ## On a DelayedArray object:
    library(HDF5Array)
    M <- as(m, "HDF5Array")
    sum2 <- block_sum(M, m_grid)
    sum2

    sum3 <- block_sum(M, colAutoGrid(M, block.length=120))
    sum3

    sum4 <- block_sum(M, rowAutoGrid(M, block.length=80))
    sum4

    ## Sanity checks:
    sum0 <- sum(m)
    stopifnot(identical(sum1, sum0))
    stopifnot(identical(sum2, sum0))
    stopifnot(identical(sum3, sum0))
    stopifnot(identical(sum4, sum0))

    ## ---------------------------------------------------------------------
    ## defaultAutoGrid()
    ## ---------------------------------------------------------------------
    grid <- defaultAutoGrid(m, block.length=120)
    grid
    as.list(grid)  # turn the grid into a list of ArrayViewport objects
    table(lengths(grid))
    stopifnot(maxlength(grid) <= 120)

    grid <- defaultAutoGrid(m, block.length=120,
                               block.shape="first-dim-grows-first")
    grid
    table(lengths(grid))
    stopifnot(maxlength(grid) <= 120)

    grid <- defaultAutoGrid(m, block.length=120,
                               block.shape="last-dim-grows-first")
    grid
    table(lengths(grid))
    stopifnot(maxlength(grid) <= 120)

    defaultAutoGrid(m, block.length=100)
    defaultAutoGrid(m, block.length=75)
    defaultAutoGrid(m, block.length=25)
    defaultAutoGrid(m, block.length=20)
    defaultAutoGrid(m, block.length=10)

    ## ---------------------------------------------------------------------
    ## rowAutoGrid() AND colAutoGrid()
    ## ---------------------------------------------------------------------
    rowAutoGrid(m, nrow=10)  # 5 blocks of 10 rows each
    rowAutoGrid(m, nrow=15)  # 3 blocks of 15 rows each plus 1 block of 5 rows
    colAutoGrid(m, ncol=5)   # 2 blocks of 5 cols each plus 1 block of 2 cols

    ## See '?write_block' for an advanced example of user-implemented
    ## block processing using colAutoGrid() and a realization sink.

    ## ---------------------------------------------------------------------
    ## REPLACE DEFAULT AUTOMATIC GRID MAKER WITH USER-DEFINED ONE
    ## ---------------------------------------------------------------------
    getAutoGridMaker()
    setAutoGridMaker(function(x) colAutoGrid(x, ncol=5))
    getAutoGridMaker()

    blockApply(m, function(block) currentViewport())

    ## Reset automatic grid maker to factory settings:
    setAutoGridMaker()

[Package DelayedArray version 0.20.0 Index]