Deriving Disease Phenotypes from UKB Data

Overview

The derive_* functions convert raw UKB columns into analysis-ready variables. This vignette covers the disease phenotype derivation pipeline:

All functions accept a data.frame or data.table and return a data.table. For data.table input, new columns are added by reference (no copy); data.frame input is converted to data.table internally before modification.

In production, replace ops_toy() with extract_batch() followed by decode_values() and decode_names(). See vignette("decode"). Column names below use the RAP raw format (p{field}_{instance}_{array}) as returned by ops_toy() and extract_batch() before decoding.

Setup

library(ukbflow)

df <- ops_toy(n = 500)

Step 1: Handle Informative Missing Labels

UKB uses special labels such as "Do not know" and "Prefer not to answer" to distinguish refusal from true missing data. derive_missing() converts these to NA (default) or retains them as "Unknown" for modelling.

df <- derive_missing(df)

Performance: derive_missing() uses data.table::set() for in-place replacement — no column copies are made regardless of dataset size.

To keep non-response as a model category:

df <- derive_missing(df, action = "unknown")

To add custom labels beyond the built-in list:

df <- derive_missing(df, extra_labels = "Not applicable")

Step 2: Prepare Covariates

derive_covariate() converts categorical columns to factor and prints a distribution summary for each.

df <- derive_covariate(
  df,
  as_factor = c(
    "p31",        # sex
    "p20116_i0",  # smoking_status_i0
    "p1558_i0"    # alcohol_intake_frequency_i0
  ),
  factor_levels = list(
    p20116_i0 = c("Never", "Previous", "Current")
  )
)

Step 3: Bin Continuous Variables

derive_cut() creates a new factor column by binning a continuous variable into quantile-based or custom groups.

df <- derive_cut(
  df,
  col    = "p21001_i0",                              # body_mass_index_bmi_i0
  n      = 4,
  breaks = c(18.5, 25, 30),
  labels = c("Underweight", "Normal", "Overweight", "Obese"),
  name   = "bmi_cat"
)

df <- derive_cut(
  df,
  col    = "p22189",                                 # townsend_deprivation_index_at_recruitment
  n      = 4,
  labels = c("Q1 (least deprived)", "Q2", "Q3", "Q4 (most deprived)"),
  name   = "tdi_cat"
)

Step 4: Self-Reported Disease

derive_selfreport() searches UKB self-reported non-cancer illness (field 20002) or cancer (field 20001) columns for a disease label matching a regex, then returns binary status and the earliest report date. Column detection is automatic from field IDs.

# Non-cancer: type 2 diabetes (field 20002)
df <- derive_selfreport(df,
  name  = "dm",
  regex = "type 2 diabetes"
)

# Cancer: lung cancer (field 20001)
df <- derive_selfreport(df,
  name  = "lung_cancer",
  regex = "lung cancer",
  field = "cancer"
)

This adds two columns per call:

Step 5: HES Inpatient Records

derive_hes() scans UKB Hospital Episode Statistics ICD-10 codes (field 41270, stored as a JSON array per participant) and matches the earliest corresponding date from field 41280.

# Prefix match: codes starting with "I10" (hypertension)
df <- derive_hes(df, name = "htn", icd10 = "I10")

# Exact match
df <- derive_hes(df, name = "dm_hes", icd10 = "E11", match = "exact")

# Regex: E10 and E11 simultaneously
df <- derive_hes(df, name = "dm_broad", icd10 = "^E1[01]", match = "regex")

The match argument controls how codes are compared:

Step 6: First Occurrence Fields

UKB First Occurrence fields (p131xxx) record the earliest date a condition was observed across all linked sources — self-report, HES inpatient, GP records, and death registry — pre-integrated by UKB. Look up your disease in the UKB Field Finder.

# ops_toy includes p131742 as a representative First Occurrence column
df <- derive_first_occurrence(df, name = "htn", field = 131742L, col = "p131742")

Step 7: Cancer Registry

derive_cancer_registry() searches the cancer registry ICD-10 field (40006) and optionally filters by histology (field 40011) and behaviour (field 40012).

# ICD-10 only
df <- derive_cancer_registry(df,
  name  = "skin_cancer",
  icd10 = "^C44"
)

# With histology and behaviour filters
df <- derive_cancer_registry(df,
  name      = "scc",
  icd10     = "^C44",
  histology = c(8070L, 8071L, 8072L),
  behaviour = 3L                        # 3 = malignant
)

Step 8: Death Registry

derive_death_registry() searches primary (field 40001) and secondary (field 40002) causes of death for ICD-10 codes.

df <- derive_death_registry(df, name = "mi",   icd10 = "I21")
df <- derive_death_registry(df, name = "dm",   icd10 = "E11")
df <- derive_death_registry(df, name = "lung", icd10 = "C34")

Step 9: Combine Sources with derive_icd10()

derive_icd10() is a high-level wrapper that calls any combination of the source-specific functions above and merges their outputs into a single status column and earliest date. This is the recommended approach for multi-source ascertainment.

# Non-cancer disease: HES + death + First Occurrence
df <- derive_icd10(df,
  name   = "dm",
  icd10  = "E11",
  source = c("hes", "death", "first_occurrence"),
  fo_col = "p131742"
)

# Cancer outcome: cancer registry
df <- derive_icd10(df,
  name      = "lung",
  icd10     = "^C3[34]",
  match     = "regex",
  source    = "cancer_registry",
  behaviour = 3L
)

Intermediate source columns are retained alongside the combined result:

Step 10: Final Case Definition

derive_case() merges the self-report and ICD-10 flags into a unified case status, with the earliest date across both sources taken via pmin().

df <- derive_case(df, name = "dm")

Output columns:

Why the earliest date matters: dm_date is the direct input to derive_timing(), derive_age(), and derive_followup() — it is the chronological anchor of every downstream survival analysis. See vignette("derive-survival").

Getting Help

• ?derive_missing, ?derive_covariate, ?derive_cut
• ?derive_selfreport, ?derive_hes, ?derive_first_occurrence
• ?derive_cancer_registry, ?derive_death_registry
• ?derive_icd10, ?derive_case
• vignette("derive-survival") — timing, age at event, follow-up
• vignette("decode") — decoding column names and values
• GitHub Issues