How much of Part VII does titleclassifier code, and how often is it right? This page reports two kinds of evidence:
- Coverage: the share of titles that reach a standard title, measured on a large stratified sample of recent filings.
- Accuracy: agreement with hand labels, broken out by form, role, pay and organization size. It is backed by checks that need no labels: agreement with the filer’s own checkboxes, and stability over time.
Every number comes from the scripts in data-raw/partvii-validation/ of the source repository, run on the NCCS efile tables (release efile_v2_3).
The data
| Data set | What it is | Used for |
|---|---|---|
| Calibration sample | 60,520 filings from 2019-2023: 6,000 organizations followed for all five years, plus 6,000 fresh organizations a year. Stratified by form (990, 990-EZ), size (total expenses, Part I) and number of paid people, with design weights | coverage; checkbox agreement; stability over time |
| Hand-labeled set | 316 people on 2010-2012 filings (276 on full 990s, 40 on 990-EZs), drawn to over-represent hard cases: paid presidents, board-run organizations, managers. Each was labeled from the whole filing in two independent passes, with disagreements settled by hand | accuracy |
| Form switchers | 23,931 people at organizations that filed a 990-EZ one year and a full 990 the next (or the reverse), matched across years with synthid. Their full-990 role, decided with the checkboxes, labels their 990-EZ year | accuracy on 990-EZ returns |
Size bins are total expenses: $0, $1-150k, $150k-1m, $1m-10m, $10m-100m and $100m+. Filers reporting $0 get their own bin because they almost never pay anyone.
Coverage
A title row is coded when step 07 finds its standard title, by exact match in the crosswalk or by a pattern rule. It is uncoded when neither applies. A few coded titles are not job titles at all (blank, “SEE SCHEDULE O”).
Weighted to the population of 2019-2023 filers, 96.0% of title rows are coded (95.2% exactly and 0.8% by pattern) and 3.8% remain uncoded. Every person still gets a role: step 09 resolves an uncoded title from the checkboxes, pay and hours.
| Filings | Title rows | Exact | Pattern | Non-job title | Uncoded | People with a role |
|---|---|---|---|---|---|---|
| All filings (sample) | 700,088 | 93.7% | 1.0% | 0.3% | 5.0% | 100.0% |
| All filings (weighted) | 700,088 | 95.2% | 0.8% | 0.2% | 3.8% | 100.0% |
| 990-EZ | 115,436 | 95.5% | 0.8% | 0.2% | 3.4% | 100.0% |
| 990 | 584,652 | 93.4% | 1.0% | 0.3% | 5.3% | 100.0% |
By size, coverage is highest for small and mid-sized organizations and lowest for the largest. Their Part VII lists many specialized staff, and their titles often include unit or system names (“HOSPITAL GROUP RETURN SCHE”, “MEMBER AND PHYSICIAN AND NGPG”):
| Form | Size | Title rows | Coded | Uncoded |
|---|---|---|---|---|
| 990 | $0 | 9,919 | 94.0% | 5.6% |
| 990 | $1-150k | 62,733 | 96.7% | 3.1% |
| 990 | $150k-1m | 151,529 | 96.6% | 3.3% |
| 990 | $1m-10m | 172,002 | 96.5% | 3.4% |
| 990 | $10m-100m | 123,947 | 93.1% | 6.8% |
| 990 | $100m+ | 64,522 | 84.1% | 14.4% |
| 990EZ | $0 | 5,674 | 94.4% | 5.4% |
| 990EZ | $1-150k | 82,115 | 96.4% | 3.4% |
| 990EZ | $150k-1m | 22,311 | 96.3% | 3.5% |
| 990EZ | $1m-10m | 1,089 | 97.0% | 2.8% |
| 990EZ | $10m-100m | 98 | 97.0% | 3.1% |
| 990EZ | missing | 4,149 | 97.8% | 1.9% |
Uncoded titles are a long tail. The 20,723 distinct uncoded titles in the sample are mostly one-offs: the 25 most common account for only 2.3% of uncoded rows. The most common are truncations, acronyms and organization names (“TAF”, “AHF”, “CLASS A POS”, “DIRECTOR CATALIGHT”), which a crosswalk entry would fix one at a time.
| Uncoded title | Rows |
|---|---|
| TAF | 62 |
| EXECUTIVE OFFICER | 51 |
| GENERAL | 47 |
| AH | 41 |
| CLASS A POS | 40 |
| AHF | 35 |
| STAFF PHYSIATRIST | 35 |
| EXPERIENCE | 35 |
| DIRECTOR CATALIGHT | 31 |
| PRINCIPAL ENGINEER AND SCIENTIST | 30 |
Accuracy
Accuracy is the share of hand-labeled people whose resolved role (role.final) matches the label. A person labeled DUAL (a board seat and a paid executive job at once) counts as correct when resolved as CEO, OFFICER or BOARD.
Overall, 89.4% of 312 labeled people get the right role. Board members on the right seat: 96.3% of correctly identified board members also get the right board role (chair, vice chair, secretary, treasurer or member).
By form
| Form | People | Accuracy | Method |
|---|---|---|---|
| 990 | 272 | 89.0% | rules, with checkboxes |
| 990EZ | 40 | 92.5% | model, no checkboxes |
On 990-EZ returns the model replaced rules that scored 82.5% on the same 40 people. On 23,931 form switchers, the model agrees with the full-990-year role more often than the rules do:
| 990-EZ people | People | Rules | Model |
|---|---|---|---|
| All matched | 23,931 | 92.0% | 95.0% |
| Same title both years | 20,817 | 95.1% | 96.6% |
| … not on the board | 1,602 | 63.7% | 73.9% |
| … paid in the 990-EZ year | 1,427 | 74.6% | 83.4% |
These scores come from the first version of the model, before the switchers were added to its training data. The shipped model, scored on the half of the switchers it did not train on, agrees 96.7% (rules 95.1%) on people with the same title in both years, and 74.0% (rules 63.6%) on those not on the board.
By role
Recall is the share of people with a given label who get that role. Precision is the share of people given a role whose label matches it.
| Role | Labeled | Recall | Resolved | Precision |
|---|---|---|---|---|
| CEO | 85 | 97.6% | 115 | 90.4% |
| OFFICER | 24 | 70.8% | 21 | 81.0% |
| MANAGER | 28 | 64.3% | 19 | 94.7% |
| PROFESSIONAL | 20 | 95.0% | 22 | 86.4% |
| STAFF | 12 | 58.3% | 19 | 36.8% |
| BOARD | 117 | 93.2% | 116 | 98.3% |
| DUAL | 26 | 100.0% |
Board members, who are most of the people on Part VII, and chief executives are identified reliably. The hard boundaries are below the CEO: officers versus managers versus staff. Those lines depend on duties that a title, pay and hours only partly reveal. STAFF is the weakest role: few people on Part VII are rank-and-file staff, and step 09 assigns STAFF to more people than are labeled so.
Checks at scale
These checks use the calibration sample and need no labels. They measure consistency: they flag problems, but agreement does not prove the answer is right.
Agreement with the filer’s checkboxes
On a full 990 the filer ticks a box for each person’s position. A board title should come with the trustee box, and a paid executive title with the officer box:
| Size | People | Board title has trustee box | Trustee box has board title | Paid executive title has officer box | Step 09 changes the title’s role |
|---|---|---|---|---|---|
| $0 | 9,099 | 80.4% | 93.9% | 79.5% | 7.7% |
| $1-150k | 60,538 | 76.9% | 94.3% | 74.5% | 4.0% |
| $150k-1m | 146,706 | 86.1% | 93.9% | 76.7% | 4.7% |
| $1m-10m | 165,668 | 94.6% | 96.0% | 79.9% | 4.1% |
| $10m-100m | 117,470 | 89.6% | 96.3% | 65.3% | 4.9% |
| $100m+ | 57,879 | 90.6% | 91.8% | 52.9% | 8.7% |
Nearly every trustee box goes with a board title (92-96% at every size). The reverse is weaker at small organizations: at $1-150k only 77% of board titles carry the trustee box, because board officers there often tick only the officer box. At $100m+ only 53% of paid executive titles carry the officer box: large organizations list many vice presidents as key employees rather than corporate officers. Step 09 treats a paid executive title as an officer position with or without the box.
Finding the chief executive
The share of filings that pay someone and in which step 09 names a CEO, and how the leadership was found:
| Form | Size | Filings | CEO named, paying filings | Designated | Imputed | Board-governed | No one paid |
|---|---|---|---|---|---|---|---|
| 990 | $0 | 1,337 | 84.2% | 21.4% | 9.9% | 4.4% | 64.3% |
| 990 | $1-150k | 8,003 | 64.2% | 13.8% | 5.8% | 7.9% | 72.5% |
| 990 | $150k-1m | 14,918 | 88.9% | 33.6% | 15.7% | 5.5% | 45.2% |
| 990 | $1m-10m | 11,427 | 95.7% | 60.7% | 19.7% | 3.5% | 16.1% |
| 990 | $10m-100m | 5,679 | 97.5% | 72.1% | 20.1% | 2.3% | 5.5% |
| 990 | $100m+ | 2,134 | 97.8% | 65.1% | 29.5% | 2.1% | 3.2% |
| 990EZ | $0 | 922 | 72.3% | 22.4% | 0.2% | 0.2% | 77.2% |
| 990EZ | $1-150k | 11,960 | 68.6% | 13.4% | 4.5% | 4.8% | 77.2% |
| 990EZ | $150k-1m | 2,885 | 82.8% | 23.4% | 11.5% | 5.8% | 59.3% |
| 990EZ | $1m-10m | 146 | 77.4% | 26.9% | 3.0% | 4.3% | 65.8% |
| 990EZ | $10m-100m | 14 | 66.7% | 12.0% | 4.0% | 4.0% | 80.1% |
| 990EZ | missing | 1,095 | 69.9% | 17.8% | 0.5% | 0.4% | 81.3% |
Among full-990 filings that pay anyone, a CEO is named in 64% at $1-150k and over 95% from $1m up. The rest are board_governed: small organizations whose only paid people are a bookkeeper or a part-time helper, or whose paid president works a few hours for a stipend.
Stability over time
For the 6,000 organizations followed for five years, people were matched from one year to the next with synthid. A person’s role should rarely change from one year to the next, and neither should the CEO:
| Form | Size | Person-years | Same role | Org-years | Same CEO |
|---|---|---|---|---|---|
| 990 | $0 | 2,468 | 96.9% | 197 | 88.3% |
| 990 | $1-150k | 20,098 | 98.0% | 795 | 92.1% |
| 990 | $150k-1m | 51,852 | 97.7% | 3,359 | 89.5% |
| 990 | $1m-10m | 58,623 | 98.4% | 3,820 | 90.6% |
| 990 | $10m-100m | 40,599 | 97.6% | 2,103 | 90.9% |
| 990 | $100m+ | 19,752 | 95.8% | 757 | 88.0% |
| 990EZ | $0 | 1,288 | 97.4% | 55 | 94.5% |
| 990EZ | $1-150k | 25,845 | 97.3% | 1,184 | 91.3% |
| 990EZ | $150k-1m | 5,292 | 95.2% | 357 | 88.8% |
| 990EZ | $1m-10m | 77 | 96.1% | 4 | 100.0% |
| 990EZ | $10m-100m | 3 | 100.0% | ||
| 990EZ | missing | 924 | 98.1% | 67 | 98.5% |
| Role in year 1 | Person-years | Same role in year 2 |
|---|---|---|
| BOARD | 188,345 | 99.1% |
| CEO | 13,167 | 91.9% |
| OFFICER | 13,040 | 90.5% |
| STAFF | 8,188 | 87.3% |
| MANAGER | 2,499 | 86.2% |
| PROFESSIONAL | 1,582 | 93.5% |
95-98% of people keep their role from one year to the next, and in most groups 88-92% of organizations keep the same CEO. Some change is real (CEOs turn over, board members become officers), so these rates are a floor on consistency, not an error rate.
Reproducing these numbers
The scripts run in order; each states its inputs at the top.
| Script | Produces |
|---|---|
12-panel-sample.R, 13-panel-run.R
|
the calibration sample and its classification |
14-panel-calibrate.R, 15-panel-synthid.R
|
checkbox agreement, CEO coverage, stability over time |
09-gold-check.R |
accuracy on the hand-labeled set |
18-switch-labels.R, 20-role-model-ship.R
|
the form-switcher test set and the shipped model |
21-performance.R |
the coverage tables and the accuracy breakdowns on this page |