Download, filter, merge, append BMF, and stack a panel

The workhorse acquisition recipe: resolve tables and years, download (or virtually scan) the source files, project columns and push the sample frame's entity restriction down to read-time, merge the tables within each year, stack the years, optionally append BMF organization traits, then apply the frame's rules. Returns a panel bundling the data with the frame and a provenance log.

Usage

panelize(
  sfw = NULL,
  tables,
  years,
  source = data_source(),
  bmf = "auto",
  backend = "memory",
  cache = c("retain", "temporary", "none"),
  path = "PANEL990",
  filters = NULL,
  columns = NULL,
  include_many = FALSE,
  collision = c("error", "prefix"),
  unique_rows = TRUE,
  overwrite = FALSE,
  retry_max = 3L,
  timeout = 1800,
  verbose = TRUE
)

Arguments

sfw: Optional create_sfw() sample frame governing the build. Filters prefilter reads and merges; select rules project columns.
tables: Aliases or literal table names.
years: Tax years.
source: Efile source configuration. Its form family decides which release is read: the default reads Form 990 and 990EZ filers; see panelize_pf() for private foundations.
bmf: Append BMF fields: "auto" (default – attach when the frame references a BMF field), FALSE, TRUE (the NCCS master), or a BMF source (data frame, path, or URL).
backend: "memory" (default) or "duckdb" ("db" is accepted).
cache: "retain", "temporary", or "none" (DuckDB-only virtual scan).
path: Retained-cache directory.
filters: Optional named value filters (merged with the frame's).
columns: Optional source fields to retain (union with the frame's).
include_many: Join one-to-many and supplemental tables.
collision: Non-key collision policy.
unique_rows: Remove exact duplicate source rows during the read. TRUE (default) preserves the historical behaviour but prevents the column projection from being pushed into a parquet scan; see read_tables().
overwrite: Replace cached files.
retry_max: Download attempts.
timeout: Minimum per-attempt download timeout in seconds. This is a budget for a whole transfer rather than an idle timeout, so it is raised automatically for large files; see download_tables().
verbose: Print progress.

Value

A panel (see as_panel()) whose data is the stacked frame, whose sfw carries the rules and provenance log (manifest()), plus download/table/join manifests and BMF diagnostics.

Details

Keys are set automatically from the efile schema (entity EIN2, time TAX_YEAR, record OBJECTID).

Reading parquet

The source format comes from source: the default data_source() reads the parquet release, and data_source(format = "csv") builds the same panel from the CSV files. Parquet pays off most for selective reads, and the frame's row and column restrictions are what make a read selective – so the combination that matters is a frame with an entity subset or a select rule, backend = "duckdb", cache = "none", and unique_rows = FALSE. See data_source() for the format's type contract and read_tables() for why deduplication blocks the projection.