Build a DuckDB database from batches of XML filings (parallel safe)

Processes XML batches in parallel, with each worker writing to its own DuckDB shard. When all workers complete, the shards are merged into a unified database for the specified year.

Usage

build_database(
  year,
  urls = NULL,
  group.size = 25,
  ccf = NULL,
  path = ".",
  is_update = FALSE
)

Arguments

year: Integer. Year label for the database.
urls: Optional character vector of XML URLs to process. If NULL, resumes from existing batch files in the year's folder.
group.size: Integer batch size (default = 25).
ccf: Concordance crosswalk object (optional).
path: Directory in which to store year subfolder and database files.
is_update: Logical; if TRUE, this is an incremental update build rather than a full-year rebuild.

Value

Invisibly returns the path to the merged DuckDB database.

Examples

if (FALSE) { # \dontrun{
build_database(2021, urls = urls, group.size = 25, path = "data")
} # }