{DBI}/{dbplyr} backend

Motivation

Connecting to Databricks SQL Warehouses typically involves using one of the methods below:

Method Advantages Disadvantages
ODBC Works with various authentication mechanisms, robust, acceptable performance.

Heavy install that can be difficult to install on corporate devices.

Very slow with larger write operations.
JDBC Not sure to be honest.

Slower than ODBC. Requires Java.

Very slow with larger write operations.
Statement Execution API Straightforward API that can handle data of all sizes. No direct support for write operations.
Databricks SQL Connector
(databricks-sql-connector)		 Convenient to work with and good performance. It’s in Python, requires {reticulate}.

That leaves a void for something that is:

{brickster} now provides both {DBI} and {dbplyr} backends for working with Databricks SQL warehouses.

It uses the Statement Execution API in combination with the Files API (the latter specifically for large data uploads).

Connecting to a Warehouse

Connecting to a Databricks SQL warehouse requires creating a {DBI} connection:

library(brickster)
library(DBI)
library(dplyr)
library(dbplyr)

# Create connection to SQL warehouse
con <- dbConnect(
  drv = DatabricksSQL(),
  warehouse_id = "<your_warehouse_id>",
  catalog = "samples"
)

{DBI} Backend

Reading Data

Execute SQL directly and return results as data frames:

trips <- dbGetQuery(con, "SELECT * FROM samples.nyctaxi.trips LIMIT 10")

Tables can be references either via Id(), I(), or using the name as-is:

# List available tables
tables <- dbListTables(con)

# Check if specific table exists
dbExistsTable(con, "samples.nyctaxi.trips")
dbExistsTable(con, I("samples.nyctaxi.trips"))
dbExistsTable(con, Id("samples", "nyctaxi", "trips"))

# Get column information
dbListFields(con, "samples.nyctaxi.trips")
dbListFields(con, I("samples.nyctaxi.trips"))
dbListFields(con, Id("samples", "nyctaxi", "trips"))

Writing Data

When writing data (append, overwrite, etc) there are two possible behaviours:

  1. In-line SQL statement write
  2. Stage .parquet files to a Volume directory and COPY INTO or CTAS

For data larger than 50k rows {brickster} will only permit the use of method (2), which requires staging_volume to be specified when establishing the connection, or directly to dbWriteQuery.

Ensure that staging_volume is a valid Volume path and you have permission to write files.

# small data (150 rows)
# generates an in-line CTAS
dbWriteTable(
  conn = con,
  name = Id(catalog = "<catalog>", schema = "<schema>", table = "<table>"),
  value = iris,
  overwrite = TRUE
)

# bigger data (4 million rows)
# writes parquet files to volume then CTAS
iris_big <- sample_n(iris, replace = TRUE, size = 4000000)

dbWriteTable(
  conn = con,
  name = Id(catalog = "<catalog>", schema = "<schema>", table = "<table>"),
  value = iris_big,
  overwrite = TRUE,
  staging_volume = "/Volumes/<catalog>/<schema>/<volume>/..." # or inherited from connection
  progress = TRUE
)

{dbplyr} Backend

Reading Data

As in the {DBI} backend tables can be referenced either via Id(), I(), or using the name as-is in tbl():

# Connect to existing tables
tbl(con, "samples.nyctaxi.trips")
tbl(con, I("samples.nyctaxi.trips"))
tbl(con, Id("samples", "nyctaxi", "trips"))
tbl(con, in_catalog("samples", "nyctaxi", "trips"))

Chain dplyr operations - they execute remotely on Databricks:

# Filter and select (translated to SQL)
long_trips <- tbl(con, "samples.nyctaxi.trips") |>
  filter(trip_distance > 10) |>
  select(
    tpep_pickup_datetime,
    tpep_dropoff_datetime,
    trip_distance,
    fare_amount
  )

# View the generated SQL (without executing)
show_query(long_trips)

# Execute and collect results
long_trips |> collect()

As a general reminder, call collect() at the latest point possible in your analysis to take reduce the required computation locally.

# Customer summary statistics
trips_summary <- tbl(con, "samples.nyctaxi.trips") |>
  group_by(pickup_zip) %>%
  summarise(
    trip_count = n(),
    total_fare_amount = sum(fare_amount, na.rm = TRUE),
    total_trip_distance = sum(trip_distance, na.rm = TRUE),
    avg_fare_amount = mean(fare_amount, na.rm = TRUE)
  ) |>
  arrange(desc(avg_fare_amount))

# Execute to get the 20 most expensive pickip zip codes with more than 30 trips
top_zipz <- trips_summary |>
  filter(trip_count > 20) |>
  head(20) |>
  collect()

Writing Data

A key difference is that temporary tables are not supported - this makes functions like copy_to only usable when specifying temporary as FALSE which will use dbWriteTable to create a table.

iris_remote <- copy_to(con, iris, "iris_table", temporary = FALSE, overwrite = TRUE)

Connection Management

The connection for the DatabricksSQL driver is different to other {DBI} backends as it doesn’t have a persistent session (it’s all just API calls). This means calling dbDisconnect serves no purpose when it comes to freeing resources on the SQL warehouse.

{WebR} Support

{bricksters} core dependencies are all {WebR} compatible. The backend uses {nanoarrow} as a fallback when {arrow} is unavailable (currently {arrow} is a Suggests).

It’s recommended to always have {arrow} installed to improve the performance of data loading.