This section covers key concepts related to the Find duplicates step.

Cluster ID

A cluster is a collection of records that have been identified as representing the same entity using the Find duplicates rules. Each cluster is identified by a unique cluster ID.

Match status/level

Each match between two records will have one of the following confidence levels:

Match status Description
Exact (0) Each individual field that makes up the record matches exactly.
Close (1) Records might have some fields that match exactly, and some fields that are very similar.
Probable (2) Records might have some fields that match exactly, some fields that are very similar, and some fields that differ a little more.
Possible (3) Records contain the majority of fields that have a number of similarities, but do not match exactly.
None (4) Records do not match.

Tags

If your data has columns tagged already, this step will recognize the tagged columns and list them as Selected columns.

This step will only recognize the following system-defined tags:

  • Company
  • Address
    • City
    • Country
    • County
    • Locality
    • Postal Code
    • Premise And Street
    • Province
    • State
    • Zip Code
  • Date
  • Email
  • Generic String
  • Phone
  • Name
    • Forenames
    • Surname
    • Title
  • Unique Id

Group IDs

You can apply different rulesets to columns with the same tag by using group IDs.

For example, you may have delivery and billing addresses that you want to treat differently. You would tag both as an address, but create separate group IDs, allowing you to apply different rulesets: only accept an exact match for the billing address, but a close one for the delivery address.

Rulesets and blocking keys

The Find duplicates step first creates blocks of similar records, which reduces the number of records which need to be compared. This is done to make the duplicate detection process more efficient. Blocks of records are created from using combinations of individual elements, to create a blocking key.

Every set of records in the resulting block is then compared using a ruleset, which is a set of logical expressions that controls the level of match which is returned.

Combinations of blocking keys and rulesets are stored in Step settings which can be selected in the Find duplicates step. To view the ruleset or blocking keys, or create a new set, go to Step settings > Find duplicates settings.

Default Find duplicates step settings

Aperture Data Studio provides default Find duplicates step settings for use with the Find duplicates step. The following default types of blocking keys and rulesets are available by navigating to Step settings > Find duplicates settings:

  • Individual - groups records with similar names at similar addresses. For example, GBR_Individual_Default will find individuals in Great Britain. Note that emails, phone numbers, and other identifiers will not be taken into account, but can be added manually.
  • Household - groups records with the same or similar family names at a similar address. For example, GBR_Household_Default will find households in Great Britain.
  • Location - groups records with similar addresses or locations. For example, GBR_Location_Default will find locations in Great Britain.

In particular, the default blocking keys and rulesets provided are for Australia (AUS), Great Britain (GBR), and United States (USA) as detailed in the table below:

Name Summary
AUS_Individual_Default Default Australia individual level rules and blocking keys based on name and address
AUS_Household_Default Default Australia household level rules and blocking keys based on surname (last name) only and address
AUS_Location_Default Default Australia location level rules and blocking keys based on address only
GBR_Individual_Default Default United Kingdom individual level rules and blocking keys based on name and address
GBR_Household_Default Default United Kingdom household level rules and blocking keys based on surname (last name) only and address
GBR_Location_Default Default United Kingdom location level rules and blocking keys based on address only
USA_Individual_Default Default United States of America individual level rules and blocking keys based on name and address
USA_Household_Default Default United States of America household level rules and blocking keys based on surname (last name) only and address
USA_Location_Default Default United States of America location level rules and blocking keys based on address only

The summary of each step setting is included to explain the purpose of the blocking keys and rulesets. The details of a step setting can also be viewed when clicked in the Step settings list screen.

Duplicate store

Retaining a duplicate store

You can retain your duplicate store to disk, so it can be used for searching and maintenance operations.

Duplicate stores are retained to your machine's Data Studio repository, within the experianmatch sub-directory. However, if you have configured a separate instance of the Find duplicates server, duplicate stores will be retained on that same machine.

To retain a duplicate store when using the Find duplicates step:

  1. Tick the Retain the duplicate store checkbox.
  2. Enter a name for your duplicate store.
  3. Click Show data to run the step and retain it to disk.

Encrypting a duplicate store

Any duplicate store can be encrypted if you specify this before running the Find duplicates step.

Encrypting the store protects data whilst the step is running on disk and is especially important for duplicate stores that have been retained for later use. Non-retained stores are deleted after the step has completed but can still be protected while the results are being processed.

To encrypt:

  1. Tick Encrypt the duplicate store in the step dialog.
  2. The encryption method will depend on whether you want to retain your store or not:
    • if you don't retain the store, a random encryption key will be generated and used;
    • if you do retain the store, a new input node will appear. You have to connect your encryption key source to this new node. The encryption key source can be as simple as a single-cell input file (with headings disabled) from the Data Explorer or a custom step that connects to a remote key vault.
  3. Click Show data to run the step and enable the encryption.