Write a mif's spatial data to disk

Materialises an in-memory mif into a self-describing directory and returns a disk-backed mif that reads from it. Metric functions then load only the columns they need, for the one sample a worker is handling, instead of the parent process holding every sample resident for the whole run.

Usage

mif_to_disk(mif, path, overwrite = FALSE)

Arguments

mif: object of class mif created with create_mif().
path: directory to write the store to. Required, with no default – a store can be tens of gigabytes, so it is always written where you say and never somewhere chosen for you. ~ is expanded.
overwrite: whether to replace an existing store at path.

Value

object of class mif whose spatial slot reads from path.

Details

Use this when the spatial data no longer comfortably fits in memory, which in practice means whole slide images. On a 283-slide cohort of 342 million cells the in-memory spatial slot is 34.2 GB against 2.19 GB of parquet, and since the parent process must hold all of it while mclapply() forks, a multi-core run cannot start at all. Disk-backing makes peak memory scale with workers rather than with the number of samples.

It is not a general speed-up. For ripleys_k() the per-worker pair list overtakes the spatial frame at roughly max(r_range) = 30, so at the default r_range the frame is a minor part of a worker's footprint; what disk-backing removes is the parent's copy of the whole cohort.

The store is a plain directory you can inspect, copy and archive:


cohort.mif/
  manifest.json      format, ids, and per-sample rows/columns/size
  clinical.rds  sample.rds
  spatial/  <sample>.parquet    one file per sample, never rewritten
  overlay/  <sample>.parquet    columns added later by split_tissue()
  derived/  <slot>.rds

Paths inside manifest.json are relative, so the whole directory can be moved or copied and still open.

What is and is not preserved

Column names, column order, types and factor levels – including unused levels and NAs – round-trip exactly, so collect_mif(mif_to_disk(x, p)) returns the spatial data x had.

The spatial slot is iterable, so sapply(mif$spatial, nrow) and lapply(mif$spatial, ...) work the same way they do on an in-memory mif. But iterating reads every sample, because lapply() materialises its input before applying anything, so peak memory becomes the whole cohort — the cost disk-backing exists to avoid. On a cohort that does not fit, iterate over names instead and only one sample is resident at a time:


# reads everything at once
sapply(mif$spatial, nrow)

# reads one sample at a time
vapply(names(mif$spatial), function(s) nrow(mif$spatial[[s]]), numeric(1))

Writing to the slot is refused outright — mif$spatial[[i]] <- x and length(mif$spatial) <- n error rather than destroying the index. Use collect_mif() to bring the data into memory if you need to modify it.

The store is shared state

A metric writes its results into the store as well as into the mif it returns, so they survive a reopen. One consequence is worth knowing, because it is the one place a disk-backed mif does not behave like an ordinary R object:


a <- ripleys_k(xd, ...)                      # result goes to `a`
identical(open_mif(store)$derived, xd$derived)   # FALSE
identical(open_mif(store)$derived, a$derived)    # TRUE

The in-memory xd is untouched, exactly as copy-on-modify implies — but xd and a point at the same directory, and the metric wrote there. The store reflects whichever mif most recently ran a metric against it.

Nothing is silently dropped: each run carries forward the derived slots it inherited, so only the slot being written is replaced. Assigning the result back to the same name (xd <- ripleys_k(xd, ...)) keeps object and store in step. To hold two parameterisations at once, give them separate stores rather than separate variables.

Row names are not preserved; they come back as 1:nrow. A columnar file has nowhere to put them, and storing them would mean an extra column per sample for something no function in this package reads – every metric indexes cells by position. Non-default row names only arise from having subset a data frame (spat[keep, ]), and dplyr resets them anyway, so this is visible only if you compare with identical() after such a subset. Use all.equal(check.attributes = FALSE), or reset them yourself, if you need that comparison to hold.

Examples

x <- create_mif(clinical_data = spatialTIME::example_clinical,
  sample_data = spatialTIME::example_summary,
  spatial_list = spatialTIME::example_spatial[1],
  patient_id = "deidentified_id", sample_id = "deidentified_sample")
store <- file.path(tempdir(), "example.mif")
xd <- mif_to_disk(x, store, overwrite = TRUE)
xd
#> 229 patients spanning 229 samples and 1 spatial data frames were found 
unlink(store, recursive = TRUE)