NeuronScope
ConceptStablesince v0.9.2

Reproducibility & Determinism Contract

The exact, bit-exact reproducibility contracts NeuronScope guarantees across models, datasets, prompts, and runtime environments — and what is explicitly non-deterministic.

Audience: AI Researchers & CI EngineersRead: 9 minEdit on GitHub

The Reproducibility Contract

In traditional AI development, silent numerical drift is the most frequent cause of invalid benchmarks, untraceable regressions, and broken production deployments. Inputs or model weights appear identical, but subtle seed variations, unpinned dependencies, or non-deterministic GPU kernel operations cause unexpected outputs.

NeuronScope turns reproducibility into a strictly enforced, verifiable software contract:

What is Guaranteed

  • Canonical Fingerprints: BLAKE3 input digests are bit-exact and stable across Linux, macOS, and Windows operating systems.
  • Unified Seed Propagation: Calling ns.seed_all(seed) synchronizes random state across Python random, NumPy, PyTorch, JAX, and CUDA generators simultaneously.
  • Deterministic Serialization: Columnar tensor streams and execution traces in .nsz artifacts are saved in a deterministic order, independent of multi-threading or async I/O.
  • Bit-Exact Evaluation Scores: Evaluation metrics calculated by ns.Evaluator yield identical values for identical telemetry inputs.

Non-Deterministic Boundaries

Modern AI hardware and cloud LLM endpoints contain inherent sources of non-determinism. NeuronScope explicitly defines and isolates these boundaries:

  • Cross-Hardware Floating Point Differences: Atomic additions in CUDA/cuDNN algorithms (e.g. reduction ops across SMs) are non-associative in float16/bfloat16. Running on an NVIDIA A100 vs. H100 GPU can produce minor numerical deltas (<1e-5). NeuronScope flags GPU architecture shifts in fp:env to distinguish hardware noise from real regressions.
  • Cloud LLM APIs (OpenAI/Anthropic): Commercial LLM API providers update backend clusters, speculative decoding models, and MoE routing dynamically. Setting temperature=0.0 reduces but does not eliminate non-determinism. NeuronScope records prompt tokens, completion digests, and vendor system fingerprints to flag API-side model shifts.

Global Seed & State Pinning

To ensure reproducible execution, use ns.seed_all() at the entry point of your training, evaluation, or inference scripts:

deterministic_pipeline.py
python
import neuronscope as ns

# 1. Enforce global determinism across Python, NumPy, PyTorch, and JAX
ns.seed_all(seed=42, deterministic_cuda=True)

# 2. Initialize Platform with strict reproducibility checks
platform = ns.Platform(
    strict_fingerprints=True,  # Rejects un-hashed datasets or models
    enforce_reproducibility=True,
)

# 3. Observe execution inside Scope
with ns.Scope("reproducible_evaluation") as scope:
    # Run evaluation benchmark
    results = ns.Evaluator.quality(model=my_model, dataset=eval_dataset)
    
# 4. Save sealed artifact
artifact = scope.save("reproducible_eval.nsz")

# 5. Assert match against baseline artifact in CI
baseline = ns.load_artifact("baseline_v1.nsz")
diff = ns.compare(baseline, artifact)

if diff.has_regression(threshold=0.001):
    raise RuntimeError(f"Reproducibility contract violated: {diff.regression_reason}")
print("✓ Bit-exact reproducibility confirmed!")

CUDA & Kernel Determinism

When deterministic_cuda=True is passed to ns.seed_all() or NS_DETERMINISTIC=1 is set in the environment, NeuronScope configures underlying ML frameworks:

  • torch.backends.cudnn.deterministic = True
  • torch.backends.cudnn.benchmark = False
  • torch.use_deterministic_algorithms(True)
  • os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"

Building a Deterministic AI Pipeline

  1. Pin Python, CUDA, and package dependencies using lockfiles (uv.lock) or container images (Dockerfile).
  2. Invoke ns.seed_all(seed=42) prior to model loading or dataset sampling.
  3. Execute Scope runs with strict fingerprint validation enabled.
  4. Store sealed reference .nsz baseline artifacts in your artifact registry or Object Store.
  5. Validate pull requests in CI pipelines using ns.compare(baseline, candidate) with automated quality gate thresholds.

Common Drift Failure Modes

  • Unordered Dict / Set Iteration: Iterating over Python sets or un-sorted dict keys introduces non-deterministic batch ordering. Always use ns.Dataset wrappers to enforce canonical sorting.
  • Multi-threaded Data Loading: PyTorch DataLoader with multiple workers without per-worker seed initializers. NeuronScope automatically injects worker seed initializers when wrapping datasets.
Related