NeuronScope
ConceptStablesince v0.4.0

Dataset & Vector Store Observability

Streaming, chunked, and vector store dataset adapters with content-addressed identity, embedding index tracking, and reproducible ordering contracts.

Audience: AI Engineers & RAG DevelopersRead: 9 minEdit on GitHub

Dataset & Vector Store Overview

In modern AI applications, datasets extend far beyond raw CSV files or JSON lines. They encompass vector database indexes (Pinecone, Qdrant, Milvus, Chroma, FAISS, Weaviate, pgvector), document chunking stores, prompt evaluation benchmarks, and streaming event payloads.

A NeuronScope Dataset is an intelligent wrapper around any data iterable or vector index. The wrapper adds content-addressed identity, schema validation, reproducible ordering contracts, and memory-bounded iterators for deterministic experiment replay.

Kinds of Dataset Adapters

NeuronScope categorizes datasets into four distinct architectural types:

  • Materialized (In-Memory): In-memory lists, Pandas DataFrames, or PyTorch Datasets. The fingerprint is derived from canonical row byte representations.
  • Chunked (Columnar): Page-based file storage (Parquet, Feather, Apache Arrow, HuggingFace Datasets). Fingerprints cover schema, column byte digests, chunk counts, and row ordering.
  • Vector Store Index: Vector database adapter (Chroma, Qdrant, Pinecone, Milvus, FAISS). Fingerprints cover vector space dimension, metric type (Cosine, L2, Dot Product), document collection schema, and document ID manifests.
  • Streaming (Generative): Continuous event streams (Kafka, SQS, custom generators). Fingerprints identify generator code identity, initial seed, and schema specification.

Vector Databases & RAG Retrieval

Retrieval quality is the single largest determinant of performance in Retrieval-Augmented Generation (RAG) applications. When querying a vector store inside a Scope, NeuronScope automatically logs:

  • Query Vectors & Text: The raw query string and optional dense/sparse embedding vectors.
  • Retrieved Document IDs: Exact document IDs returned from the index in order.
  • Similarity Scores: Distance scores (cosine similarity, inner product) for each returned document chunk.
  • Retrieval Latency: Wall-clock time spent in vector search and index traversal.
vector_store_demo.py
python
import neuronscope as ns
import chromadb

# 1. Wrap a vector database collection (ChromaDB)
client = chromadb.Client()
collection = client.get_collection("enterprise_knowledge_base")
vector_index = ns.Dataset.wrap_vector_store(collection)

print(f"Vector Store Name:  {vector_index.name}")
print(f"Vector Fingerprint: {vector_index.fingerprint()}")
print(f"Metric & Dim:       {vector_index.metric} | {vector_index.dimension}d")

# 2. Observe RAG retrieval query inside Scope
with ns.Scope("rag_evaluation") as scope:
    # Query vector store — automatically captures query vector, top_k document IDs, and scores
    results = vector_index.query(query_texts=["Explain zero-copy deserialization"], top_k=3)
    scope.log_retrieval(query="Explain zero-copy deserialization", docs=results)

# 3. Save sealed artifact with full retrieval provenance
artifact = scope.save("rag_eval.nsz")
print(f"Retrieved {len(artifact.retrieval_spans)} search spans")

Dataset Fingerprinting Engine

How does NeuronScope compute a fast fp:dataset:... fingerprint on a 50GB Parquet dataset without scanning every byte on every run?

The platform uses a Hierarchical Merkle Tree (BLAKE3) approach:

  • Chunk-level hashes are computed asynchronously or read directly from Parquet metadata block checksums.
  • The root hash is formed by hashing the ordered array of chunk digests along with dataset metadata (schema, total rows, features).
  • This yields an O(1) cached fingerprint for unchanged files and O(N/chunks) for modified datasets.
huggingface_parquet.py
python
import neuronscope as ns
from datasets import load_dataset

# Load HuggingFace dataset or Apache Arrow / Parquet table
hf_dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="test")

# Wrap as NeuronScope Dataset — computes BLAKE3 chunk fingerprints lazily
ds = ns.Dataset.from_huggingface(hf_dataset, name="wikitext_test")

print(f"Dataset Name:        {ds.name}")
print(f"Total Rows:          {len(ds)}")
print(f"Canonical Digest:    {ds.fingerprint()}")

# Deterministic reproducible batching with pinned random seed
dataloader = ds.batch(batch_size=32, shuffle=True, seed=42)
for batch_idx, batch in enumerate(dataloader):
    # Process batch...
    if batch_idx == 0:
        print(f"First Batch Fingerprint: {ns.fingerprint(batch)}")

Reproducible Ordering & Seeds

Data order directly impacts model training loss trajectories and evaluation benchmark scores. NeuronScope enforces reproducible ordering through:

  • Canonicalization: Unordered input collections (e.g. Python set or un-sorted SQL queries) are automatically sorted by key/hash before fingerprinting.
  • Pinned Random Seeds: Shuffling operations accept a mandatory seed parameter. The random generator state is bound to the dataset scope.

Streaming & Memory Boundaries

Multi-Source Data Pipelines

When an AI system combines multiple data sources (e.g., user query history + vector search context + SQL relational metadata), ns.Dataset.combine() generates a composite dataset fingerprint (fp:dataset_group:...). This guarantees that changes to any component dataset trigger regression warnings in the comparison engine.

Related