Fingerprint a Hugging Face dataset

Hugging Face datasets are often streamed, sharded, and lazily downloaded. Fingerprinting them naïvely would either fail (streaming iterators are single-shot) or take minutes. This recipe uses the schema-and-generator hash documented inthe fingerprinting chapter.

Problem

You want a fingerprint over the train split of a HF dataset that is stable across machines and does not download the whole thing.

Recipe

import neuronscope as ns
from datasets import load_dataset

hf = load_dataset("wikitext", "wikitext-2-raw-v1", split="train", streaming=True)

# ns.Dataset.from_hf uses the streaming schema + generator identity,
# so nothing is materialized on disk.
ds = ns.Dataset.from_hf(hf, name="wikitext-2-raw-v1:train")

print(ds.fingerprint)
# fp:dataset:1e8f...

Expected output

$ python fingerprint_hf.py
fp:dataset:1e8fa79ca4b4d8c5e1f0a2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e6f7
copy

Notes

• The fingerprint depends on the split's schema and the loader identity — not on the currently downloaded shard set. That is the trade-off documented in ADR-011.
• Non-streaming splits fingerprint over materialized rows and match across machines only if the row order is stable — pass ordered=True to the HF loader.