Mehdi Akiki
Published on

Entity Resolution Before Machine Learning: Exact, Normalized, and Reviewed Matches

Authors
  • Mehdi Akiki avatar
    Name
    Mehdi Akiki
    Twitter

Reference

Entity resolution asks whether two records refer to the same real-world entity. It is tempting to begin with embeddings or a probabilistic model.

I begin with identity scope, exact evidence, and reversible decisions. These layers often solve a large share of records and create the labels needed to evaluate anything more complex.

Similar is not the same

These contacts look similar:

Mehdi Akiki, [email protected]
M. Akiki, [email protected]

Shared verified email may be strong evidence in one product. In another, the address is a shared team inbox and proves little.

These records look different:

ACME SARL
Acme Morocco Ltd.

They may be the same legal entity after a name change, or separate subsidiaries.

A similarity score cannot define identity without a domain rule.

Start with scoped durable identifiers

I inventory identifiers and their namespaces:

IdentifierScopeReuse riskTrust source
provider contact IDprovider accountprovider-specificprovider API
verified emailoften many systemsshared/recycledverification event
tax registrationjurisdiction and entity typelow but not zerogoverned registry
phone numbercountry/timerecycled/sharedverification event
internal canonical IDproduct modelcontrolledidentity service

An exact match on an unscoped provider ID is not exact identity. It is an accidental collision.

The canonical-ID article A Canonical ID Is an Internal Promise, Not a Provider ID covers the mapping boundary.

Normalize one field according to its meaning

I do not put every value through one generic lowercase-and-remove-punctuation function.

Examples:

email: domain case rules, provider aliases only when justified
phone: parse with country context into an international form
company name: legal suffix dictionary by jurisdiction
address: structured country-specific components
Unicode name: normalization without inventing transliteration equivalence

Normalization should be reversible or retain the original. Two raw values producing the same normalized value become candidates, not automatically the same entity.

For Gmail-like plus aliases, applying one provider's rule to every email domain can merge unrelated mailboxes.

Use deterministic rules with evidence levels

I define match rules such as:

R1 high: same scoped immutable provider identity
R2 high: same governed registration ID and jurisdiction
R3 medium: same verified email plus compatible entity type
R4 candidate only: normalized name + postal code
R5 candidate only: phone suffix + similar address

Each decision stores which rule fired and the source values at that time.

type MatchDecision = {
  leftRecordId: string;
  rightRecordId: string;
  decision: "match" | "not_match" | "review";
  ruleOrModelVersion: string;
  evidenceReferences: string[];
  score?: number;
  decidedAt: string;
  decidedBy: "rule" | "reviewer" | "model";
};

This makes later audit and unmerge possible.

Blocking controls the comparison space

Comparing every pair among one million records requires roughly half a trillion pairs. Most are obviously unrelated.

Blocking creates plausible candidate sets:

same normalized email domain and name prefix
same postcode and company-name token
same phone country and subscriber number
same external account scope

I use several blocking keys so a missing field in one key does not make a true match impossible. Candidate recall is measured separately from final match precision.

If the correct pair never enters the candidate set, a perfect scoring model cannot recover it.

Negative evidence matters

Two similar names with incompatible facts should not merge:

different governed registration IDs
person versus company entity type
simultaneously active provider accounts known to be separate
countries inconsistent with a supposedly exact local identifier
explicit prior reviewer rejection

I encode hard contradictions before soft similarity. A model score should not overrule a trusted non-match constraint silently.

Review the uncertain middle

I use three outcomes:

automatic match above a precision-tested threshold
automatic non-match below a threshold
human review in the ambiguous interval

The review interface shows field-level agreements, differences, provenance, prior decisions, and what a merge would affect. It does not show only “92% match.”

Review decisions create useful labels, but they can contain bias and inconsistency. I sample agreement between reviewers and adjudicate difficult classes.

Add a model after the baseline is measurable

A probabilistic or learned model can help combine weak signals and spelling variation. Before deploying it, I compare against deterministic layers on a representative labelled set.

I report:

candidate-generation recall
automatic-match precision
automatic-match recall
false merges by business impact
false splits by business impact
review volume and time
performance by country, language, provider, and entity type

False merges are often more costly than false splits because they mix private data, balances, or entitlements. The operating threshold reflects that asymmetry.

A merge must be reversible

The resolver proposes identity. A separate workflow applies it with version checks and an audit trail.

propose mapping
validate no hard conflicts
preview affected records and permissions
apply versioned merge
rebuild derived projections
retain aliases and source provenance
support split/unmerge

I avoid rewriting all historical source IDs. They are evidence needed for provider synchronization and repair.

An incorrect merge may be discovered months later, so decision inputs and model/rule versions need appropriate retention.

Avoid feature leakage in evaluation

If the labelled match was created because two records already shared a canonical ID, using that canonical ID as a model feature makes evaluation perfect and meaningless.

I split training and evaluation by entity and time, not random record pairs only. Otherwise near-duplicate pairs for the same entity can appear in both sets.

I also test changes in provider format and missing-field patterns, because production drift often appears there first.

Reconcile identity continuously

New evidence can contradict an old mapping. I monitor:

  • one governed ID attached to several canonical entities;
  • a canonical entity with mutually exclusive attributes;
  • a provider identity remapped without a decision event;
  • high-confidence rules whose false-merge sample grows;
  • review backlog age;
  • model-score drift by source;
  • merge and split propagation failures.

Entity resolution is not a one-time deduplication batch. It is a maintained product boundary.

The strongest starting system is often not the most complex model. It is a scoped identifier inventory, field-specific normalization, high-precision rules, broad candidate blocking, explicit negative evidence, and a review loop that can explain and undo every identity decision.