link_records {diyar}R Documentation

Record linkage

Description

Deterministic and probabilistic record linkage with partial or evaluated matches.

Usage

link_records(
  attribute,
  blocking_attribute = NULL,
  cmp_func = diyar::exact_match,
  attr_threshold = 1,
  probabilistic = TRUE,
  m_probability = 0.95,
  u_probability = NULL,
  score_threshold = 1,
  repeats_allowed = FALSE,
  permutations_allowed = FALSE,
  data_source = NULL,
  ignore_same_source = TRUE,
  display = "none"
)

links_wf_probabilistic(
  attribute,
  blocking_attribute = NULL,
  cmp_func = diyar::exact_match,
  attr_threshold = 1,
  probabilistic = TRUE,
  m_probability = 0.95,
  u_probability = NULL,
  score_threshold = 1,
  id_1 = NULL,
  id_2 = NULL,
  ...
)

prob_score_range(attribute, m_probability = 0.95, u_probability = NULL)

Arguments

attribute

[atomic|list|data.frame|matrix|d_attribute]. Attributes to compare.

blocking_attribute

[atomic]. Subsets of the dataset.

cmp_func

[list|function]. String comparators for each attribute. See Details.

attr_threshold

[list|numeric|number_line]. Weight-thresholds for each cmp_func. See Details.

probabilistic

[logical]. If TRUE, scores are assigned base on Fellegi-Sunter model for probabilistic record linkage. See Details.

m_probability

[list|numeric]. The probability that a matching records are the same entity.

u_probability

[list|numeric]. The probability that a matching records are not the same entity.

score_threshold

[numeric|number_line]. Score-threshold for linked records. See Details.

repeats_allowed

[logical] If TRUE, repetition are included.

permutations_allowed

[logical] If TRUE, permutations are included.

data_source

[character]. Data source identifier. Adds the list of data sources in each record-group to the pid. Useful when the data is from multiple sources.

ignore_same_source

[logical] If TRUE, only records from different data_source are compared.

display

[character]. Display or produce a status update. Options are; "none" (default), "progress", "stats", "none_with_report", "progress_with_report" or "stats_with_report".

id_1

[list|numeric]. Record id or index of one half of a record pair.

id_2

[list|numeric]. Record id or index of one half of a record pair.

...

Arguments passed to links

Details

link_records() and links_wf_probabilistic() are functions to implement deterministic, fuzzy or probabilistic record linkage. link_records() compares every record-pair in one instance, while links_wf_probabilistic() is a wrapper function of links and so compares batches of record-pairs in iterations.

link_records() is more thorough in the sense that it compares every combination of record-pairs. This makes it faster but is memory intensive, particularly if there's no blocking_attribute. In contrast, links_wf_probabilistic() is less memory intensive but takes longer since it does it's checks in batches.

The implementation of probabilistic record linkage is based on Fellegi and Sunter (1969) model for deciding if two records belong to the same entity.

In summary, record-pairs are created and categorised as matches and non-matches (attr_threshold) with user-defined functions (cmp_func). Two probabilities (m and u) are then estimated for each record-pair to score the matches and non-matches. The m-probability is the probability that matched records are actually from the same entity i.e. a true match, while u-probability is the probability that matched records are not from the same entity i.e. a false match. By default, u-probabilities are calculated as the frequency of each value of an attribute however, they can also be supplied along with m-probabilities. Record-pairs whose total score are above a certain threshold (score_threshold) are assumed to belong to the same entity.

Agreement (match) and disagreement (non-match) scores are calculated as described by Asher et al. (2020).

For each record pair, an agreement for attribute i is calculated as;

\log_{2}(m_{i}/u_{i})

For each record pair, a disagreement score for attribute i is calculated as;

\log_{2}((1-m_{i})/(1-u_{i}))

where m_{i} and u_{i} are the m and u-probabilities for each value of attribute i.

Note that each probability is calculated as a combined probability for the record pair. For example, if the values of the record-pair have u-probabilities of 0.1 and 0.2 respectively, then the u-probability for the pair will be 0.02.

Missing data (NA) are considered non-matches and assigned a u-probability of 0.

By default, matches and non-matches for each attribute are determined as an exact_match with a binary outcome. Alternatively, user-defined functions (cmp_func) are used to create similarity scores. Pairs with similarity scores within (attr_threshold) are then considered matches for the corresponding attribute.

If probabilistic is FALSE, the sum of all similarity scores is used as the score_threshold instead of deriving one from the m and u-probabilities.

A blocking_attribute can be used to reduce the processing time by restricting comparisons to subsets of the dataset.

In link_records(), score_threshold is a convenience argument because every combination of record-pairs are returned therefore, a new score_threshold can be selected after reviewing the final scores. However, in links_wf_probabilistic(), the score_threshold is more important because a final selection is made at each iteration.

As a result, links_wf_probabilistic() requires an acceptable score_threshold in advance. To help with this, prob_score_range() can be used to return the range of scores attainable for a given set of attribute, m and u-probabilities. Additionally, id_1 and id_2 can be used to link specific records pairs, aiding the review of potential scores.

Value

pid; list

References

Fellegi, I. P., & Sunter, A. B. (1969). A Theory for Record Linkage. Journal of the Statistical Association, 64(328), 1183–1210. https://doi.org/10.1080/01621459.1969.10501049

Asher, J., Resnick, D., Brite, J., Brackbill, R., & Cone, J. (2020). An Introduction to Probabilistic Record Linkage with a Focus on Linkage Processing for WTC Registries. International journal of environmental research and public health, 17(18), 6937. https://doi.org/10.3390/ijerph17186937.

See Also

links

Examples

# Deterministic linkage
dfr <- missing_staff_id[c(2, 4, 5, 6)]

link_records(dfr, attr_threshold = 1, probabilistic = FALSE, score_threshold = 2)
links_wf_probabilistic(dfr, attr_threshold = 1, probabilistic = FALSE,
                       score_threshold = 2, recursive = TRUE)

# Probabilistic linkage
prob_score_range(dfr)
link_records(dfr, attr_threshold = 1, probabilistic = TRUE, score_threshold = -16)
links_wf_probabilistic(dfr, attr_threshold = 1, probabilistic = TRUE,
                       score_threshold = -16, recursive = TRUE)

# Using string comparators
# For example, matching last word in `hair_colour` and `branch_office`
last_word_wf <- function(x) tolower(gsub("^.* ", "", x))
last_word_cmp <- function(x, y) last_word_wf(x) == last_word_wf(y)

link_records(dfr, attr_threshold = 1,
             cmp_func = c(diyar::exact_match,
                          diyar::exact_match,
                          last_word_cmp,
                          last_word_cmp),
             score_threshold = -4)
links_wf_probabilistic(dfr, attr_threshold = 1,
                    cmp_func = c(diyar::exact_match,
                                 diyar::exact_match,
                                 last_word_cmp,
                                 last_word_cmp),
                    score_threshold = -4,
                    recursive = TRUE)


[Package diyar version 0.4.1 Index]