Getting started: a panel walk-through

What `synthid` does

The upstream packages (titleclassifier, peopleparser) answer “what is this record?” — a parsed name, a standardized title. synthid answers a different question: “who is this record, across years?”

Given a stacked, multi-year panel of Form 990 director/officer/trustee compensation rows, it assigns every record a stable per-person identifier, EMP_ID, that is consistent across tax years. That identifier is the keystone a longitudinal panel design rests on: the raw person.id in the source data is not stable across years, so a dedicated linkage layer is required.

This article walks the whole pipeline on a tiny synthetic panel you can paste and run.

The input: a compensation panel

synthid consumes a data frame with, at minimum, an organization id (ein), a tax year (taxyr), and a full name. If the parsed name components and a standardized title are present, they are used to sharpen the match. Here is a small panel with three organizations-worth of behavior baked in:

panel <- rbind(
  data.frame(ein = "100", taxyr = 2019, name = "JOHN SMITH SR",
             first_name = "JOHN", last_name = "SMITH", suffix = "SR",
             gender = "M", title.standard = "CEO"),
  data.frame(ein = "100", taxyr = 2019, name = "JOHN SMITH JR",
             first_name = "JOHN", last_name = "SMITH", suffix = "JR",
             gender = "M", title.standard = "TREASURER"),
  data.frame(ein = "100", taxyr = 2019, name = "ANN JONES",
             first_name = "ANN", last_name = "JONES", suffix = "",
             gender = "F", title.standard = "SECRETARY"),
  data.frame(ein = "100", taxyr = 2020, name = "JOHN SMITH SR",
             first_name = "JOHN", last_name = "SMITH", suffix = "SR",
             gender = "M", title.standard = "CEO"),
  data.frame(ein = "100", taxyr = 2020, name = "JOHN SMITH JR",
             first_name = "JOHN", last_name = "SMITH", suffix = "JR",
             gender = "M", title.standard = "TREASURER"),
  data.frame(ein = "100", taxyr = 2021, name = "JOHN SMITH SR",
             first_name = "JOHN", last_name = "SMITH", suffix = "SR",
             gender = "M", title.standard = "CEO"),
  # A different organization with a same-named person -- must NOT cross-link.
  data.frame(ein = "200", taxyr = 2019, name = "JOHN SMITH SR",
             first_name = "JOHN", last_name = "SMITH", suffix = "SR",
             gender = "M", title.standard = "PRESIDENT"),
  data.frame(ein = "200", taxyr = 2020, name = "JOHN SMITH SR",
             first_name = "JOHN", last_name = "SMITH", suffix = "SR",
             gender = "M", title.standard = "PRESIDENT"),
  stringsAsFactors = FALSE
)
panel
#>   ein taxyr          name first_name last_name suffix gender title.standard
#> 1 100  2019 JOHN SMITH SR       JOHN     SMITH     SR      M            CEO
#> 2 100  2019 JOHN SMITH JR       JOHN     SMITH     JR      M      TREASURER
#> 3 100  2019     ANN JONES        ANN     JONES             F      SECRETARY
#> 4 100  2020 JOHN SMITH SR       JOHN     SMITH     SR      M            CEO
#> 5 100  2020 JOHN SMITH JR       JOHN     SMITH     JR      M      TREASURER
#> 6 100  2021 JOHN SMITH SR       JOHN     SMITH     SR      M            CEO
#> 7 200  2019 JOHN SMITH SR       JOHN     SMITH     SR      M      PRESIDENT
#> 8 200  2020 JOHN SMITH SR       JOHN     SMITH     SR      M      PRESIDENT

There are three deliberate challenges here:

A father/son pair on the same board (JOHN SMITH SR and JOHN SMITH JR) who share a first and last name and differ only by suffix. They must stay two people.
A person observed across three years (JOHN SMITH SR, org 100) who should collapse to one id spanning 2019-2021.
A same-named person at a different organization (JOHN SMITH SR, org 200) who must not be linked to the org-100 Smiths.

Link the panel

One call does the whole pipeline — prepare, block, compare, score, match, and close the accepted links into person clusters:

linked <- link_panel(panel)
#> Warning: Ignoring feature(s) not present in data: salutation, middle_name
linked[, c("ein", "taxyr", "name", "suffix", "EMP_ID", "EMP_N_RECORDS", "EMP_N_YEARS")]
#>   ein taxyr          name suffix                                      EMP_ID
#> 1 100  2019 JOHN SMITH SR     SR EMP-D4B7-4F7F-14F5-B552-70D2-8A29-BF62-A46E
#> 2 100  2019 JOHN SMITH JR     JR EMP-A7CE-0F8E-4131-43EB-6D11-89C4-189E-6037
#> 3 100  2019     ANN JONES        EMP-25E6-D5BB-9E4A-51E8-A248-0213-8B81-957D
#> 4 100  2020 JOHN SMITH SR     SR EMP-D4B7-4F7F-14F5-B552-70D2-8A29-BF62-A46E
#> 5 100  2020 JOHN SMITH JR     JR EMP-A7CE-0F8E-4131-43EB-6D11-89C4-189E-6037
#> 6 100  2021 JOHN SMITH SR     SR EMP-D4B7-4F7F-14F5-B552-70D2-8A29-BF62-A46E
#> 7 200  2019 JOHN SMITH SR     SR EMP-03FD-E4A8-F1FD-E858-F107-88A8-1D98-8ADE
#> 8 200  2020 JOHN SMITH SR     SR EMP-03FD-E4A8-F1FD-E858-F107-88A8-1D98-8ADE
#>   EMP_N_RECORDS EMP_N_YEARS
#> 1             3           3
#> 2             2           2
#> 3             1           1
#> 4             3           3
#> 5             2           2
#> 6             3           3
#> 7             2           2
#> 8             2           2

Read the added columns:

EMP_ID — the stable person identifier (a content hash of the cluster’s members, so it is reproducible across runs).
EMP_N_RECORDS — how many records are in the person’s cluster.
EMP_N_YEARS — how many distinct tax years the person is observed.

Check that the three challenges resolved correctly:

# The org-100 SR appears in 3 years under ONE id.
sr100 <- linked$EMP_ID[linked$ein == "100" & linked$suffix == "SR"]
length(unique(sr100)) == 1L
#> [1] TRUE

# Father and son (SR vs JR) are DIFFERENT people.
jr100 <- linked$EMP_ID[linked$ein == "100" & linked$suffix == "JR"]
unique(sr100) != unique(jr100)
#> [1] TRUE

# The same-named person at org 200 is NOT the org-100 person.
sr200 <- linked$EMP_ID[linked$ein == "200"]
length(intersect(sr100, sr200)) == 0L
#> [1] TRUE

The run report

synthid_report() prints a summary of what happened — how many clusters formed, how many span multiple years, and how many links were blocked by the one-record-per-organization-year guard:

synthid_report(linked)
#> synthid linkage report
#> ----------------------
#> records            : 8
#> organizations      : 2
#> years              : 2019, 2020, 2021
#> person clusters    : 4
#>   multi-year       : 3
#>   singletons       : 1
#> accepted links     : 5
#> collisions blocked : 0
#> method             : weighted (score >= 7)
#> panel compression  : 50.0% (records -> persons)

How it decided (in one paragraph)

Candidate pairs are formed only within an organization and only across different tax years — blocks are small, so every cross-year pair inside an org is compared. Each field contributes weight x (2*similarity - 1); a missing field is neutral. Two touches matter for the panel above: the surname weight is scaled per organization by how rare the surname is inside that org, so agreeing on a common board surname carries little information and the suffix / first name / gender do the discriminating (this is what keeps SR and JR apart); and because pairs are only formed within an ein, the org-200 Smith is never even compared to the org-100 Smiths. Accepted links are then closed into clusters under a hard invariant: a person holds at most one record per organization-year.

A learned alternative: the EM model

The default method = "weighted" uses a hand-set additive score. You can instead fit an unsupervised Fellegi-Sunter latent-class model to the candidate comparisons and link by the learned posterior match probability — no labels required. It also attaches EMP_LINK_CONF, the weakest link probability holding each cluster together:

linked_em <- link_panel(panel, method = "em")
#> Warning: Ignoring feature(s) not present in data: salutation, middle_name
#> Warning: Ignoring feature(s) not present in data: salutation, middle_name
linked_em$EMP_LINK_CONF
#> [1]  1  1 NA  1  1  1  1  1

# The learned agreement / disagreement weights:
fs_weights(attr(linked_em, "synthid")$model)
#>          feature     m     u w_agree w_disagree
#> 1     first_name 1.000 0.000   13.29     -13.29
#> 2      last_name 1.000 0.000   13.29     -13.29
#> 3         suffix 0.556 0.000   12.44      -1.17
#> 4         gender 1.000 0.000   13.29     -13.29
#> 5 title.standard 0.556 0.333    0.74      -0.58

The EM fit doubles as a validation of the hand weights — on the real development panel the two methods agree on 99.85% of accept/reject decisions — and the posterior is sharply bimodal, which is what makes EMP_LINK_CONF a usable confidence signal for a downstream panel model.

Adapting to your column names

If your panel names the organization or other fields differently, pass a modified column map. You can also raise the threshold for higher precision:

cols <- synthid_cols()
cols$org_id <- "ein"   # e.g. change to "ein9" for a 9-digit EIN column
str(cols)
#> List of 5
#>  $ org_id  : chr "ein"
#>  $ org_name: chr "org.name"
#>  $ year    : chr "taxyr"
#>  $ name    : chr "name"
#>  $ features: chr [1:7] "salutation" "first_name" "middle_name" "last_name" ...

# linked <- link_panel(panel, cols = cols, threshold = 8)

Where the identifiers come from

EMP_ID answers who, across years. There is a second identifier, PERSON_YEAR_ID, that answers which source row — a stable back-link from a linked record to its original Form 990 Part VII line, computed from the source key at ingest. The two are complementary: EMP_ID groups a person’s rows together; PERSON_YEAR_ID ties each of those rows back to the raw filing it came from. See the function reference for person_year_id() and create_emp_ids().

Where to go next

The function reference, grouped by subsystem (panel linkage, identifiers, match model, name comparison, cross-organization linkage, title roles, and link review).
link_panel() for the full set of arguments (method, threshold, weights, prob_threshold).
default_weights() and fs_weights() to inspect and tune the scoring. ```

What synthid does