Dataset Overlap Analysis | Eidogen-Sertanty

Two-panel diagram. Panel one: a compound, with its assay, primary target and potency, is reduced by a one-way SHA-256 hash to an irreversible encoded fingerprint, and the original structure is kept private. Panel two: two parties each hold a private list of encoded fingerprints, compare them, and learn only the percentage that overlaps. — Illustrative example. Figures shown are for explanation only and are not results from our datasets.

How it works

1. Each side encodes its own data, locally. Every datapoint is reduced to a single SHA-256 fingerprint over five normalized fields — structure (InChIKey), target (UniProt accession), assay measure, value and unit. The tool makes no network calls; your data never leaves your machine.

2. Only fingerprints are exchanged. The encoded file is a list of 64-character hexadecimal digests. There are no structures, target names, or assay values in it.

3. The fingerprints are compared. Matching digests are shared datapoints; everything else is novel to one side. The report gives counts and percentages — and nothing about the other party's holdings.

Works with any target family

Kinases, GPCRs, ion channels, proteases, nuclear receptors, or any mixture. No target list is built into the tools. If your data already carries UniProt accessions it works immediately; if it carries gene symbols, build_target_map.py resolves them for any family you name — and reports anything it could not resolve rather than guessing.

The two tools that touch your data are plain Python 3 with no dependencies and no install step, so your security team can read every line before you run it.

Try it in 30 seconds

The download includes a sample dataset, so you can run the whole pipeline end to end before pointing it at anything of your own — nothing to configure, no network access, no account.

unzip novelty_assessment_tools.zip
cd novelty_assessment

# encode each side separately - in a real exchange these run
# on two different machines, at two different organizations
python3 encode_dataset.py sample_data/sample_setA.csv -o A_encoded.csv
python3 encode_dataset.py sample_data/sample_setB.csv -o B_encoded.csv

# compare the fingerprints
python3 compare_datasets.py --reference A_encoded.csv \
                            --vendor B_encoded.csv -o report.html

Expected output, exactly:

Vendor datapoints:   3,955
Novel:               1,582 (40.0%)
Overlap:             2,373 (60.0%)

If you get those three numbers, your install is correct. The sample is the published Kinase Validation Set — 3,955 activities for ABL1, SRC and AURKA — paired with a constructed second set that shares a known 60% of its rows, so the answer is checkable.

Download the toolkit → Sample data only → Talk to us about a comparison →

A note on what this protects

Encoded files contain only one-way digests, and all processing of your data is local. But the five hashed fields are drawn from a small space, and only the structure key carries real entropy — so an encoded file should be treated as sensitive and exchanged with a counterparty you have an agreement with, not published openly. For comparisons that need to stay sound against a motivated counterparty, ask us about the private set intersection protocol, where neither side ever holds an unkeyed hash of the other's data.