Skip to contents

How much of Part VII does titleclassifier code, and how often is it right? This page reports two kinds of evidence:

  • Coverage: the share of titles that reach a standard title, measured on a large stratified sample of recent filings.
  • Accuracy: agreement with hand labels, broken out by form, role, pay and organization size. It is backed by checks that need no labels: agreement with the filer’s own checkboxes, and stability over time.

Every number comes from the scripts in data-raw/partvii-validation/ of the source repository, run on the NCCS efile tables (release efile_v2_3).

The data

Data set What it is Used for
Calibration sample 60,520 filings from 2019-2023: 6,000 organizations followed for all five years, plus 6,000 fresh organizations a year. Stratified by form (990, 990-EZ), size (total expenses, Part I) and number of paid people, with design weights coverage; checkbox agreement; stability over time
Hand-labeled set 316 people on 2010-2012 filings (276 on full 990s, 40 on 990-EZs), drawn to over-represent hard cases: paid presidents, board-run organizations, managers. Each was labeled from the whole filing in two independent passes, with disagreements settled by hand accuracy
Form switchers 23,931 people at organizations that filed a 990-EZ one year and a full 990 the next (or the reverse), matched across years with synthid. Their full-990 role, decided with the checkboxes, labels their 990-EZ year accuracy on 990-EZ returns
CEO label set 750 filings from 2019-2023: 150 organizations over five years, in 14 strata chosen where the chief executive is hard to find (unions, clubs, lodges, veterans posts, universities, ministries, fire departments, multi-CEO systems…) plus ordinary charities. Two independent labelers gave every row a role and board role, and named each filing’s chief executive; the scores use the rows where they agree chief executive and board role, by kind of organization
Held-out CEO set 126 filings (2021 and 2023) from 64 other organizations of the 8 types the CEO thesaurus covers, labeled the same way. The thesaurus was written before they were labeled; one threshold was adjusted after the same, on organizations the rules were not written from

Size bins are total expenses: $0, $1-150k, $150k-1m, $1m-10m, $10m-100m and $100m+. Filers reporting $0 get their own bin because they almost never pay anyone.

Coverage

A title row is coded when step 07 finds its standard title, by exact match in the crosswalk or by a pattern rule. It is uncoded when neither applies. A few coded titles are not job titles at all (blank, “SEE SCHEDULE O”).

Weighted to the population of 2019-2023 filers, 96.1% of title rows are coded (95.3% exactly and 0.8% by pattern) and 3.7% remain uncoded. Every person still gets a role: step 09 resolves an uncoded title from the checkboxes, pay and hours.

Filings Title rows Exact Pattern Non-job title Uncoded People with a role
All filings (sample) 699,276 93.8% 1.0% 0.3% 4.9% 100.0%
All filings (weighted) 699,276 95.3% 0.8% 0.2% 3.7% 100.0%
990-EZ 115,408 95.6% 0.8% 0.2% 3.4% 100.0%
990 583,868 93.5% 1.0% 0.3% 5.2% 100.0%

By size, coverage is highest for small and mid-sized organizations and lowest for the largest. Their Part VII lists many specialized staff, and their titles often include unit or system names (“HOSPITAL GROUP RETURN SCHE”, “MEMBER AND PHYSICIAN AND NGPG”):

Form Size Title rows Coded Uncoded
990 $0 9,905 94.6% 5.1%
990 $1-150k 62,726 96.8% 3.0%
990 $150k-1m 151,470 96.7% 3.2%
990 $1m-10m 171,860 96.5% 3.4%
990 $10m-100m 123,651 93.3% 6.6%
990 $100m+ 64,256 84.4% 14.1%
990EZ $0 5,666 94.4% 5.4%
990EZ $1-150k 82,105 96.5% 3.3%
990EZ $150k-1m 22,305 96.2% 3.5%
990EZ $1m-10m 1,085 96.9% 2.9%
990EZ $10m-100m 98 97.0% 3.1%
990EZ missing 4,149 97.7% 2.0%

Uncoded titles are a long tail. The 20,723 distinct uncoded titles in the sample are mostly one-offs: the 25 most common account for only 2.3% of uncoded rows. The most common are truncations, acronyms and organization names (“TAF”, “AHF”, “CLASS A POS”, “DIRECTOR CATALIGHT”), which a crosswalk entry would fix one at a time.

Uncoded title Rows
TAF 62
GENERAL 45
AH 41
CLASS A POS 40
AHF 35
EXPERIENCE 35
DIRECTOR CATALIGHT 31
REGIONAL YOR 30
DIRECTOR “” 29
TRUSTEE AND REGIONAL POINTE 29

Accuracy

Accuracy is the share of hand-labeled people whose resolved role (role.final) matches the label. A person labeled DUAL (a board seat and a paid executive job at once) counts as correct when resolved as CEO, OFFICER or BOARD.

Overall, 90.4% of 312 labeled people get the right role. Board members on the right seat: 98.2% of correctly identified board members also get the right board role (chair, vice chair, secretary, treasurer or member).

By form

Form People Accuracy Method
990 272 90.1% rules, with checkboxes
990EZ 40 92.5% model, no checkboxes

On 990-EZ returns the model replaced rules that scored 82.5% on the same 40 people. On 23,931 form switchers, the model agrees with the full-990-year role more often than the rules do:

990-EZ people People Rules Model
All matched 23,931 92.0% 95.0%
Same title both years 20,817 95.1% 96.6%
… not on the board 1,602 63.7% 73.9%
… paid in the 990-EZ year 1,427 74.6% 83.4%

These scores come from the first version of the model, before the switchers were added to its training data. The shipped model, scored on the half of the switchers it did not train on, agrees 96.7% (rules 95.1%) on people with the same title in both years, and 74.0% (rules 63.6%) on those not on the board.

By role

Recall is the share of people with a given label who get that role. Precision is the share of people given a role whose label matches it.

Role Labeled Recall Resolved Precision
CEO 85 100.0% 115 92.2%
OFFICER 24 70.8% 21 81.0%
MANAGER 28 64.3% 19 94.7%
PROFESSIONAL 20 95.0% 22 86.4%
STAFF 12 58.3% 18 38.9%
BOARD 117 94.0% 117 98.3%
DUAL 26 100.0%

Board members, who are most of the people on Part VII, and chief executives are identified reliably. The hard boundaries are below the CEO: officers versus managers versus staff. Those lines depend on duties that a title, pay and hours only partly reveal. STAFF is the weakest role: few people on Part VII are rank-and-file staff, and step 09 assigns STAFF to more people than are labeled so.

By pay

Paid by the organization People Accuracy
paid 224 87.5%
unpaid 88 97.7%

Unpaid people are almost all board members, and nearly always classified correctly. Paid people carry the hard cases: is a paid president the CEO, is a paid director a board member or a department head?

By size

Size People Accuracy
$1-150k 61 86.9%
$150k-1m 84 96.4%
$1m-10m 99 92.9%
$10m-100m 58 82.8%
$100m+ 10 80.0%

The labeled set has no $0 filers and only 10 people at $100m+ organizations, so the size breakdown is indicative only. The checks below cover every size at scale.

Checks at scale

These checks use the calibration sample and need no labels. They measure consistency: they flag problems, but agreement does not prove the answer is right.

Agreement with the filer’s checkboxes

On a full 990 the filer ticks a box for each person’s position. A board title should come with the trustee box, and a paid executive title with the officer box:

Size People Board title has trustee box Trustee box has board title Paid executive title has officer box Step 09 changes the title’s role
$0 9,099 80.5% 93.8% 79.6% 7.8%
$1-150k 60,538 77.0% 94.1% 74.4% 4.3%
$150k-1m 146,706 86.2% 93.8% 76.8% 4.8%
$1m-10m 165,668 94.8% 96.0% 80.0% 4.1%
$10m-100m 117,470 90.1% 96.5% 65.4% 4.6%
$100m+ 57,879 91.5% 92.1% 53.0% 8.1%

Nearly every trustee box goes with a board title (92-96% at every size). The reverse is weaker at small organizations: at $1-150k only 77% of board titles carry the trustee box, because board officers there often tick only the officer box. At $100m+ only 53% of paid executive titles carry the officer box: large organizations list many vice presidents as key employees rather than corporate officers. Step 09 treats a paid executive title as an officer position with or without the box.

Finding the chief executive

The share of filings that pay someone and in which step 09 names a CEO, and how the leadership was found:

Form Size Filings CEO named, paying filings Designated Imputed Board-governed No one paid
990 $0 1,337 84.2% 21.4% 9.9% 4.4% 64.3%
990 $1-150k 8,003 64.3% 13.8% 5.8% 7.9% 72.5%
990 $150k-1m 14,918 88.9% 33.6% 15.7% 5.5% 45.2%
990 $1m-10m 11,427 95.7% 60.7% 19.7% 3.5% 16.1%
990 $10m-100m 5,679 97.5% 72.1% 20.1% 2.3% 5.5%
990 $100m+ 2,134 97.8% 65.1% 29.5% 2.1% 3.2%
990EZ $0 922 75.4% 22.3% 0.0% 0.2% 75.0%
990EZ $1-150k 11,960 70.8% 13.2% 0.0% 4.5% 76.9%
990EZ $150k-1m 2,885 85.0% 23.0% 0.0% 5.1% 59.2%
990EZ $1m-10m 146 83.9% 26.2% 0.0% 3.0% 63.8%
990EZ $10m-100m 14 66.7% 12.0% 0.0% 4.0% 80.1%
990EZ missing 1,095 72.2% 17.6% 0.0% 0.4% 81.1%

Among full-990 filings that pay anyone, a CEO is named in 64% at $1-150k and over 95% from $1m up. The rest are board_governed: small organizations whose only paid people are a bookkeeper or a part-time helper, or whose paid president works a few hours for a stipend.

The chief executive, by kind of organization

Naming a CEO is not the same as naming the right one. The CEO label sets test that on 2019-2023 filings, where the labelers chose each filing’s chief executive (or none, for a board-run organization).

The two labelers named the same chief executive for 98.4% of the 750 filings and 96.9% of the held-out ones.

Organizations Filings Right CEO Board role right Held out: filings Held out: right CEO
All 750 88.1% 98.7% 126 81.7%
ministry 50 96.0% 99.4% 14 57.1%
orchestra 50 96.0% 99.3%
trade association 50 94.0% 98.0%
club or co-op 50 92.0% 100.0% 16 75.0%
lodge 50 92.0% 97.5% 16 87.5%
museum 50 92.0% 99.8%
benefit trust or cemetery 50 90.0% 98.8% 16 100.0%
fire department 50 90.0% 99.2% 16 87.5%
university 50 90.0% 98.2% 16 62.5%
veterinary association 50 88.0% 97.6%
control charity 100 86.0% 98.4%
multi-CEO system 50 82.0% 99.8%
union 50 82.0% 96.5% 16 87.5%
veterans post 50 66.0% 98.4% 16 93.8%

Step 09 names the right chief executive for 88.1% of the 750 filings and for 81.7% of the held-out ones. Board seats get the right board role 98.7% of the time.

Two rounds of changes were made against the 750 filings:

  1. Rules for stipends, past presidents and unpaid full-time leaders raised the right-CEO share from 80.9% to 85.1%.
  2. The CEO thesaurus raised it to 87.6%.
  3. Letting a better-paid university president outrank a paid chancellor (F-034) raised it to 88.1%.

On the held-out filings the thesaurus raised it from 79.4% to 81.7%, all of it in lodges (68.8% to 87.5%).

Veterans posts are the hardest of the 750. Their adjutants, quartermasters and commanders split the work, and often the labelers name no CEO at all. Unions come next: the one in charge may be a business manager, a paid president or a secretary-treasurer.

In the held-out set, ministries and universities are the weakest, each with only 14-16 filings: * a ministry run by two people, such as a married couple; * an unpaid religious president of a college; * a president who reports one hour a week while a chancellor is listed.

Stability over time

For the 6,000 organizations followed for five years, people were matched from one year to the next with synthid. A person’s role should rarely change from one year to the next, and neither should the CEO:

Form Size Person-years Same role Org-years Same CEO
990 $0 2,468 96.9% 197 88.3%
990 $1-150k 20,098 98.1% 795 92.2%
990 $150k-1m 51,852 97.7% 3,363 89.6%
990 $1m-10m 58,623 98.4% 3,818 90.6%
990 $10m-100m 40,599 97.6% 2,103 90.9%
990 $100m+ 19,752 95.8% 757 87.7%
990EZ $0 1,288 98.3% 60 95.0%
990EZ $1-150k 25,845 97.9% 1,190 91.3%
990EZ $150k-1m 5,292 97.0% 364 90.7%
990EZ $1m-10m 77 97.4% 4 100.0%
990EZ $10m-100m 3 100.0%
990EZ missing 924 97.5% 68 97.1%
Role in year 1 Person-years Same role in year 2
BOARD 189,518 99.2%
OFFICER 13,788 89.8%
CEO 13,248 91.7%
STAFF 6,214 89.4%
MANAGER 2,343 87.5%
PROFESSIONAL 1,710 93.6%

95-98% of people keep their role from one year to the next, and in most groups 88-92% of organizations keep the same CEO. Some change is real (CEOs turn over, board members become officers), so these rates are a floor on consistency, not an error rate.

Reproducing these numbers

The scripts run in order; each states its inputs at the top.

Script Produces
12-panel-sample.R, 13-panel-run.R the calibration sample and its classification
14-panel-calibrate.R, 15-panel-synthid.R checkbox agreement, CEO coverage, stability over time
09-gold-check.R accuracy on the hand-labeled set
18-switch-labels.R, 20-role-model-ship.R the form-switcher test set and the shipped model
21-performance.R the coverage tables and the accuracy breakdowns on this page
24-ceo-sample.R to 27-ceo-rerun.R the CEO label sets (CEO_SET=ceo-holdout for the held-out one): sample, blind case files, scores