Document frequency of each name token across the reference, turned into
idf = log(N / df): rare tokens (distinctive, high IDF) are the ones the
name-token blocking leverages; corpus-common tokens (low IDF) are not. Feed
the result to npmatch::np_route() (token_idf =) to annotate the tokenized name
fields with each token's rarity, exposing which tokens carry the match.
Usage
np_token_idf(name_keys, stopwords = np_stopwords(), min_token_len = 2L)Arguments
- name_keys
Character vector of normalized names (reference
name_key).- stopwords
Tokens to drop (see
np_stopwords()).- min_token_len
Minimum token length to count. Default 2.