Python SDK — SDK — NeuronScope

SDK Overview

The neuronscope Python package is the primary developer surface for the NeuronScope AI Engineering Platform. It provides a unified interface for observing, evaluating, profiling, explaining, optimizing, and governing AI systems throughout their complete lifecycle.

Unlike traditional AI monitoring libraries that focus on a single aspect — logging, metrics, or dashboards — NeuronScope treats the entire AI system as a first-class engineering object. Every model call, retrieval pass, agent step, tool invocation, and workflow execution is captured, fingerprinted, and organized into a structured telemetry artifact.

The SDK covers the complete AI engineering lifecycle:

Observe: Collect real-time telemetry across every component of your AI system.
Evaluate: Measure quality, hallucination rate, latency, and cost using standardized pipelines.
Profile: Identify performance bottlenecks in GPU memory, token consumption, and latency.
Explain: Generate interpretable reasoning paths, prompt influence maps, and context attribution.
Compare: Diff two runs, models, or deployments to detect regressions and improvements.
Govern: Enforce RBAC, audit logging, PII masking, and enterprise security policies.
Improve: Use operational knowledge to continuously optimize future deployments.

Installation & Extras

Install the full platform SDK or pick only the extras you need:

terminal

bash

pip install "neuronscope[all]==0.9.2"
# or install with specific extras:
pip install "neuronscope[torch,llm]==0.9.2"

Available installation extras:

[torch] — PyTorch adapter, layer probes, GPU VRAM profiling, activation capture.
[jax] — JAX/Flax adapter with functional transformation tracing.
[tf] — TensorFlow/Keras adapter for layer extraction and gradient hooks.
[llm] — OpenAI, Anthropic, Cohere, Mistral, and Ollama API instrumentation.
[vector] — Pinecone, Qdrant, Chroma, Milvus, Weaviate vector store adapters.
[agents] — LangChain, LlamaIndex, AutoGen, and custom agent execution tracing.
[cloud] — S3, GCS, Azure Blob artifact persistence and streaming.
[all] — Complete suite: every adapter, evaluator, and governance engine.

Public API Surface

Every top-level export from import neuronscope as ns:

surface.py

python

import neuronscope as ns

# ── Core Platform ────────────────────────────────────────────────────
ns.Platform       # Central platform initialization & service discovery
ns.Scope          # Non-invasive AI observability context manager
ns.Evaluator      # Hallucination, accuracy, relevance & quality evaluators
ns.Profiler       # Token rate, VRAM, latency, and throughput profiler
ns.Governance     # Enterprise RBAC, audit logging, and policy engine

# ── Data Primitives ──────────────────────────────────────────────────
ns.Prompt         # Structured, hashable prompt object with variable binding
ns.Dataset        # Dataset and vector store index wrapper with streaming
ns.Model          # Model wrapper providing non-invasive probe attachment
ns.Agent          # Agent wrapper for tool-call and reasoning-path tracing

# ── Fingerprinting & Identity ────────────────────────────────────────
ns.fingerprint    # BLAKE3 canonical fingerprinting — models, prompts, datasets
ns.Environment    # Captured runtime: Python, OS, CUDA, BLAS, seeds

# ── Telemetry & Artifacts ────────────────────────────────────────────
ns.Experiment     # Recorded run with pinned inputs, telemetry & portable artifact
ns.load_artifact  # Open and inspect sealed .nsz telemetry files
ns.compare        # Regression diffing & comparison engine across runs

# ── Probes & Hooks ───────────────────────────────────────────────────
ns.probes         # Layer activation probes, attention maps, gradient hooks
ns.hooks          # Tool-call hooks, API span hooks, agent decision hooks

Platform Initialization

The ns.Platform class is the central coordination engine for NeuronScope. It initializes the AI Kernel, registers adapters, configures the storage layer, and sets up security policies. Most users never instantiatePlatform directly — it is configured once via YAML or environment variables.

platform.py

python

import neuronscope as ns

# Minimal: uses defaults from neuronscope.yaml or NS_* env vars
platform = ns.Platform()

# Explicit configuration
platform = ns.Platform(
    storage_path="./ns_artifacts",
    log_level="info",
    max_overhead_pct=15,       # abort if observability overhead > 15%
    security_policy="strict",  # enforce RBAC and audit logging
)

# Platform is a context manager
with ns.Platform() as platform:
    # All SDK calls within this block use this platform instance
    env = ns.Environment.current()
    print(f"Platform ready — environment: {env.fingerprint()}")

On startup, Platform discovers all registered adapters via the Plugin Framework, validates configuration, captures the current Environment, and prepares the telemetry serialization pipeline.

Scope — AI Observability

ns.Scope is the primary observability primitive. It attaches non-invasive probes to any AI component — LLMs, embedding models, RAG pipelines, agents, workflow nodes — and records comprehensive telemetry into a portable .nsz artifact.

scope.py

python

import neuronscope as ns

# Observe an LLM API call end-to-end
with ns.Scope("gpt4_query", monitor_all=True) as scope:
    response = openai_client.chat.completions.create(
        model="gpt-4-turbo",
        messages=[{"role": "user", "content": "Explain AI observability"}]
    )

# Inspect what was captured
print(scope.summary())
# → tokens_in=12  tokens_out=187  latency_ms=1240  cost_usd=0.0042
scope.save("gpt4_query.nsz")

Key Scope Parameters

name — Human-readable experiment name stored in the artifact header.
monitor_all — Auto-instrument all detected AI subsystems in the current context.
out — Destination path for the sealed .nsz artifact. None keeps it in memory.
deterministic — Refuses to run if any installed op is known non-deterministic.
max_overhead_pct — Aborts recording if observability overhead exceeds this percentage.
tags — Arbitrary string key-value metadata embedded in the artifact for filtering.

Evaluator — AI Evaluation

ns.Evaluator provides standardized evaluation pipelines for measuring AI system quality. Evaluation runs during development, testing, staging, and production — giving engineering teams continuous quality signals across the full deployment lifecycle.

evaluator.py

python

import neuronscope as ns

artifact = ns.load_artifact("rag_run.nsz")

# Evaluate hallucination using retrieved context
result = ns.Evaluator.hallucination(
    artifact,
    method="context_attribution",  # or "nli", "selfcheck", "perplexity"
    threshold=0.15,
)

print(f"Hallucination score: {result.score:.}")  # 0 = grounded, 1 = hallucinated
print(f"Status: {result.status}")  # PASS or FAIL
print(f"High-risk spans: {result.flagged_spans}")

Profiler — Performance Analysis

ns.Profiler collects detailed performance telemetry across CPU, GPU, memory, token consumption, and workflow execution. Profiling runs automatically within any active Scope, or can be invoked independently for targeted analysis.

profiler.py

python

import neuronscope as ns

with ns.Scope("llm_query") as scope:
    response = model.generate(prompt)

# Extract performance profile from scope
profile = ns.Profiler.from_scope(scope)

print(f"Total latency:     {profile.total_ms:.} ms")
print(f"Time to  token: {profile.ttft_ms:.} ms")
print(f"Token throughput:  {profile.tokens_per_sec:.} tok/s")
print(f"Peak VRAM:         {profile.peak_vram_mb:.} MB")
print(f"CPU utilization:   {profile.cpu_pct:.}%")
print(f"Total tokens:      {profile.total_tokens} (in={profile.tokens_in}, out={profile.tokens_out})")
print(f"Estimated cost:    ${profile.cost_usd:.6f}")

Fingerprinting & Identity

ns.fingerprint generates deterministic, portable BLAKE3-based identities for models, prompts, datasets, environments, and complete AI systems. Fingerprints are the primitive every other NeuronScope guarantee is built upon — reproducibility, regression detection, and change tracking all rely on them.

fingerprint.py

python

import neuronscope as ns
from transformers import AutoModel

model = AutoModel.from_pretrained("meta-llama/Llama-2-")

# Compute deterministic fingerprint over model weights
fp = ns.fingerprint(model)
print(fp.arch)     # fp:arch:a3c1f8…72  (architecture fingerprint)
print(fp.weights)  # fp:wt:9a0b2d…4f   (weight digest — changes on fine-tuning)
print(fp.full)     # fp:full:d4e7b9…11  (complete model fingerprint)

# Fingerprints are stable across Python versions and platforms
# Same model weights → same fingerprint, guaranteed

Prompt & Context Management

ns.Prompt treats prompts as first-class, hashable engineering objects — not strings. Every prompt template, variable binding, and rendered output is tracked, fingerprinted, and stored as part of the telemetry artifact.

prompt.py

python

import neuronscope as ns

# Create a versioned, structured prompt
prompt = ns.Prompt(
    template="You are a {role}. Answer based on: {context}\n\nQuestion: {question}",
    version="v2.1.0",
    variables={
        "role": "helpful AI assistant",
        "context": retrieved_docs,
        "question": user_query,
    },
)

# Render and fingerprint
rendered = prompt.render()
print(prompt.fingerprint())  # Changes when template or variables change

# Track prompt evolution across experiments
with ns.Scope("prompt_experiment") as scope:
    scope.track_prompt(prompt)
    response = llm.generate(rendered)

Dataset & Vector Store

ns.Dataset wraps training, evaluation, and retrieval datasets with deterministic fingerprinting, streaming support, and vector store integration. It provides reproducible data ordering and change detection across the full AI data pipeline.

dataset.py

python

import neuronscope as ns

# Load and fingerprint a local dataset
dataset = ns.Dataset.load("eval_data.jsonl")
print(dataset.fingerprint())   # Deterministic identity
print(dataset.schema)          # Inferred schema
print(len(dataset))            # Row count

# Load from Hugging Face Hub
from datasets import load_dataset
hf = load_dataset("squad", split="validation")
dataset = ns.Dataset.from_hf(hf)

# Load a vector store for RAG evaluation
vector_ds = ns.Dataset.from_vector_store(
    pinecone_index,
    embedding_model="text-embedding-3-small",
)

Comparison & Regression Diffing

ns.compare diffs two telemetry artifacts, models, prompts, or datasets to detect regressions, improvements, and structural changes. The Regression Comparer Engine performs layer-wise, step-by-step alignment using canonical fingerprints — telling you exactly which component changed and by how much.

compare.py

python

import neuronscope as ns

baseline  = ns.load_artifact("baseline_v1.nsz")
candidate = ns.load_artifact("candidate_v2.nsz")

diff = ns.compare(baseline, candidate)

# Summary of all changes
print(diff.summary())

# Check for regressions
if diff.has_regression(threshold=0.05):
    print(f"REGRESSION: {diff.regression_reason}")
    print(f"  Quality delta:     {diff.quality_delta:+.3f}")
    print(f"  Hallucination D:   {diff.hallucination_delta:+.3f}")
    print(f"  Latency P95 D ms:  {diff.latency_p95_delta_ms:+.0f}")
    print(f"  Token cost D:      {diff.token_cost_delta:+.4f}")

# Step-level breakdown
for step in diff.steps:
    print(f"  {step.name}: latency_delta={step.latency_delta_ms:+.0f}ms")

Governance & Security

ns.Governance provides enterprise-grade security controls, audit logging, PII masking, and deployment approval workflows. Security is integrated at every layer — not added as an afterthought.

governance.py

python

import neuronscope as ns

# Initialize platform with enterprise security policy
platform = ns.Platform(
    security_policy="rbac",
    rbac_config={
        "require_role": ["engineer", "researcher"],
        "audit_all_requests": True,
        "pii_masking": True,
        "allowed_models": ["gpt-4", "claude-3", "llama-*"],
    }
)

# Check governance before deployment
gov = ns.Governance(platform)
report = gov.validate_deployment(
    artifact="candidate_v2.nsz",
    policy="production_policy.yaml",
)

if not report.approved:
    print(f"Deployment blocked: {report.blocking_reasons}")
else:
    print(f"Deployment approved — audit ID: {report.audit_id}")

Complete Engineering Workflow

The following example demonstrates the complete NeuronScope engineering workflow — from development through production deployment — using the Python SDK:

complete_workflow.py

python

import neuronscope as ns

# ── 1. Initialize Platform ───────────────────────────────────────────
platform = ns.Platform(storage_path="s3://my-bucket/ns-artifacts")

# ── 2. Load Dataset & Model ──────────────────────────────────────────
eval_dataset    = ns.Dataset.load("rag_eval_queries.jsonl")
test_queries    = eval_dataset.sample(n=100, seed=42)

# ── 3. Observe & Profile Production Run ───────────────────────────────
with ns.Scope("rag_production_v3", tags={"stage": "production"}) as scope:
    for query in test_queries:
        docs = vector_store.search(query, k=5)
        scope.log_retrieval(query=query, docs=docs)
        answer = my_rag_model.generate(build_prompt(query, docs))

# ── 4. Evaluate Quality ───────────────────────────────────────────────
eval_result = ns.Evaluator.rag(
    scope,
    metrics=["hallucination", "answer_faithfulness", "retrieval_precision"],
)
print(f"Hallucination: {eval_result.hallucination:.}")
print(f"Faithfulness:  {eval_result.answer_faithfulness:.3f}")

# ── 5. Profile Performance ────────────────────────────────────────────
profile = ns.Profiler.from_scope(scope)
print(f"P95 latency:   {profile.p95_latency_ms:.0f}ms")
print(f"Token cost:    ${profile.cost_usd:.4f} per query")

# ── 6. Compare Against Baseline ──────────────────────────────────────
baseline  = ns.load_artifact("rag_production_v2.nsz")
candidate = scope.artifact
diff = ns.compare(baseline, candidate)

if diff.has_regression(threshold=0.03):
    raise SystemExit(f"Regression detected: {diff.regression_reason}")
print(f"Improvement: {diff.quality_delta:+.3f}")

# ── 7. Seal & Archive Artifact ────────────────────────────────────────
scope.save("rag_production_v3.nsz")
print("Artifact sealed and archived.")

# ── 8. Generate Governance Report ────────────────────────────────────
gov = ns.Governance(platform)
report = gov.validate_deployment("rag_production_v3.nsz")
print(f"Governance: {report.status}  Audit ID: {report.audit_id}")

API Stability & SemVer

NeuronScope follows Semantic Versioning 2.0.0 for all top-level neuronscope module exports.

Major version: Breaking changes to public API signatures, artifact format, or fingerprint semantics.
Minor version: New capabilities, new evaluation metrics, new adapter support — fully backward compatible.
Patch version: Bug fixes, performance improvements, documentation updates.

All private API surfaces (prefixed with _), internal engine classes, and plugin-facing APIs carry@experimental or @beta decorators indicating their stability guarantees.