Roll a parse-fail log up into candidate title/credential tokens

Reduces the record-level parse_fail_log() to the small, structured list a peopleparser maintainer actually acts on: the distinct tokens that leaked into a name field, one row each, with frequency and an example. Tokens are recovered structurally from the log's last_name/expected_last and first_name/expected_first columns (the set-difference of the wrong parse and the expected value) – no text parsing of the evidence string.

Usage

parse_fail_tokens(log, known = character())

Arguments

log

A data frame from parse_fail_log().

known

Optional character vector of already-known tokens (e.g. toupper(peopleparser::known_titles()$abbr)); each candidate is marked in_known. Tokens with in_known == FALSE are the add-list.

Value

A data frame, one row per (token, field): token, field ("surname" or "given"), defect_type, n_records, n_persons, example (a raw name showing it), and in_known. Ordered by in_known (novel first) then n_records descending.

Details

The credential_in_surname rows yield the post-nominal tokens that belong in the prefix/suffix strip list and known_titles(); the honorific_glue rows yield the honorific that failed to split (typically already known – the defect is tokenisation, not a missing token, so in_known will be TRUE). surname_letter_glue contributes nothing (a stray initial is not a title).

See also

parse_fail_log()

Examples

if (FALSE) { # \dontrun{
log <- parse_fail_log(linked)
# pass the live peopleparser list as `known` to flag which tokens are novel:
# known <- toupper(peopleparser::known_titles()$abbr)
toks <- parse_fail_tokens(log, known = c("MR", "DR", "MD", "PHD"))
subset(toks, !in_known)          # the tokens to add upstream
} # }