Package index • ef2

The concordance

The master crosswalk mapping XML XPaths to standardized variable names and relational tables, plus tools to refresh and validate it.

get_concordance(): Retrieve the Concordance File

prep_concordance(): Prepare a concordance crosswalk (uppercase colnames)

update_concordance(): Refresh the packaged concordance from GitHub

concordance_is_current(): Check whether the packaged concordance is current

The filing index

Locate the returns available to process from the Giving Tuesday Data Commons.

get_current_index_full(): Load the full IRS 990 e-filer index from the Data Commons

get_current_index_batch(): Load the most recent IRS 990 e-filer batch index from the Data Commons

find_current_index_full(): Find Most Recent AWS Full Index

find_current_index_batch(): Find Most Recent AWS Batch Index

download_current_index_full(): Download Current AWS Index

Index helpers

Lower-level helpers used to resolve and validate index URLs.

url_is_valid(): Validate URL Status

get_url_status(): Get URL Status

get_url_status_df(): Get URL Status for Multiple Days

get_last_n_dates(): Get Last N Dates

extract_dates(): Extract Dates from Filenames

find_most_recent_date(): Find Most Recent Date

get_index_list_awscli(): Get AWS Index List

extract_filenames_full(): Extract Index Filenames

extract_filenames_batch(): Extract Batch Index Filenames

get_current_index_full_awscli(): Get Current AWS Index Using CLI

get_all_batch_indices_awscli(): Get All Batch Index Filenames Using CLI

Creating batch files

Split an index of URLs into batches for parallel processing.

create_batchfiles(): Create batchfiles (RDS) for multiple years

split_index(): Split index to batchfile RDS for a single year

prep_index(): Prepare an index for batching

split_urls(): Split a URL vector into named groups and persist RDS

split_into_groups(): Utility: split a vector into labeled groups

write_batches(): Write batch files to disk

gather_batches(): Load pending batch files from disk

remove_batch(): Remove a processed batch file

Flattening XML

Convert one nested 990 XML filing into a long, one-row-per-node table.

flatten_xml(): Flatten an IRS 990 XML document to long-form rows

get_flat_xml(): Download, parse, and flatten a single XML filing (with retries)

batch_flatten(): Flatten a batch of XML filings and write to DuckDB

get_attr_df(): Extract all XML node attributes into a tidy data frame

get_keys(): Get table keys for a single filing

XML node & XPath helpers

Utilities used during flattening to classify nodes, derive table ids, and read values from the XML tree.

get_type(): Classify xpaths as parent or terminal

find_parent_nodes(): Find parent node xpaths for a set of xpaths

find_terminal_nodes(): Find terminal node xpaths for a set of xpaths

get_header(): Compute the TABLE_HEADER from an xpath

get_table_id(): Make a TABLE_ID from a vector of xpaths

get_n(): Extract the last bracketed index from an xpath

get_vnames(): Vectorized variable name extraction from xpaths

get_xpath_vname(): Get the last node name from an xpath

get_object_id(): Extract an OBJECTID from a filing URL

get_object_id2(): Extract an OBJECTID from a filing URL

retrieve_xml(): Extract text from XML nodes

standardize_boole(): Standardize boolean inputs

namedList(): Create a named list from its arguments

Writing to DuckDB

Persist flattened filings into DuckDB tables.

send_flat_xml_to_db(): Write flattened XML batch results to an existing DuckDB connection

check_for_columns(): Ensure destination DuckDB table has all required columns

Building the database

Orchestrate parallel flattening of batches into a merged DuckDB database.

build_database(): Build a DuckDB database from batches of XML filings (parallel safe)

resume_build_database(): Resume a partial DuckDB build

merge_duckdbs(): Merge multiple worker DuckDB databases into a main database

validate_merge(): Validate a merged DuckDB database against its worker sources

Updating the database

Detect and process filings missing from an existing database.

find_missing_urls(): Identify missing URLs in a given tax year

update_db(): Update the DuckDB database for a given tax year

merge_databases(): Merge DuckDB databases with schema alignment and timestamped logfile

Connecting to DuckDB & S3

Open DuckDB connections and read/write databases on S3.

open_database(): Open a DuckDB database connection with S3 support

get_s3_database(): Attach an S3-hosted DuckDB database by filename

configure_aws_credentials(): Configure AWS credentials for DuckDB session

write_csv_to_s3(): Write a DuckDB table to CSV on S3 via COPY

Extracting tables

Pivot the flattened data into analysis-ready one-to-one and one-to-many relational tables.

extract_csv_tables(): Extract All IRS 990 Tables from DuckDB Databases

build_table(): Build a structured wide table and optionally export to CSV/S3

build_rdb_table(): Build an RDB table from multiple header variants

flatten_table(): Flatten a logical RDB table into wide format from FLATXML

add_keys(): Add KEYS columns to a flattened table

get_table_names(): Get RDB table names from the concordance file

get_table_headers(): Get table headers

Inspecting & reporting

Summarize database contents and report on XPath coverage across filings.

inspect_ddb(): Inspect a DuckDB database interactively

summarize_attr_table(): Summarize attribute structure from a DuckDB database

summarize_attr_schema(): Summarize the attribute schema from a tidy ATTRIBUTES table

retrieve_attr_df(): Retrieve attribute data from a DuckDB database

generate_xpath_report(): Generate an XPATH Summary Report from a DuckDB Database

process_xpaths(): Combine and Process Multi-Year XPATH Reports

Exploring table structure

Diagram the hierarchical structure of a table from its concordance XPaths.

get_table_xpaths(): Get Table XPaths

get_nd(): Get Node Tree

create_edgelist_v1(): Create Edgelist Version 1

create_edgelist_v2(): Create Edgelist Version 2

print_table_str(): Print Table Structure

plot_table_str(): Plot Table Structure

Utilities

General-purpose helpers.

format_ein(): Format Employer Identification Numbers (EINs)

extract_functions(): Extract Function Names and Arguments from an R Script

generate_ascii_diagram(): Generate an ASCII Diagram of Functions and Arguments from R Scripts

Data

Datasets and data-structure references shipped with the package.

concordance: Concordance of IRS e-file XPaths to standardized variables and tables

index: IRS 990 e-filer index (structure reference)