System Comparison & Regression Diffing

Why System Comparison Matters

Modern AI applications are continuously evolving. Models are fine-tuned. Prompts are revised. RAG pipelines gain new retrieval steps. Vector stores are re-indexed. Workflow nodes are reordered. Each change has the potential to improve — or degrade — system quality, latency, and reliability.

Traditional software has mature tools for comparing code changes: diffs, test suites, and regression checks. AI systems require a different approach. Unlike deterministic software, AI systems produce probabilistic outputs that depend on model weights, retrieval quality, prompt formulation, and runtime environment.

Comparing AI systems requires answering questions like:

• Which component introduced the quality regression — the model, the retrieval, or the prompt?
• Did the new fine-tuned model hallucinate more or less than the baseline?
• Which workflow step became the new latency bottleneck after the infrastructure upgrade?
• Did the new prompt template improve answer faithfulness without increasing token cost?
• Is the candidate deployment safe to promote to production?

The NeuronScope Regression Comparer Engine was built to answer all of these questions systematically — using canonical fingerprints, step-level telemetry alignment, and structured quality metrics.

How the Comparison Engine Works

Comparison operates on sealed .nsz telemetry artifacts produced byns.Scope. Each artifact captures a complete fingerprinted record of one AI system execution — including model weights, prompt variables, retrieval results, execution traces, evaluation scores, profiling data, and environment metadata.

The Comparison Engine performs three phases:

• Phase 1 — Structural Alignment: Match execution steps between baseline and candidate using canonical layer IDs and fingerprints. Detect added, removed, or reordered components.
• Phase 2 — Metric Computation: Compute quality deltas, latency regressions, hallucination drift, and cost variance across aligned steps.
• Phase 3 — Regression Decision: Apply configurable thresholds to determine whether the candidate represents a regression, improvement, or neutral change.

Canonical Layer & Step Alignment

Before metrics are computed, the Comparer aligns the baseline and candidate execution traces step-by-step using canonical layer IDs and prompt fingerprints. This alignment is deterministic and handles:

• Added steps: New retrieval passes, new tool calls, or new workflow nodes in the candidate.
• Removed steps: Eliminated workflow branches or skipped retrieval stages.
• Reordered steps: Steps executing in a different sequence due to workflow restructuring.
• Structural breaks: Different model architectures, incompatible prompt templates, or environment changes that prevent step-level alignment.

import neuronscope as ns

baseline  = ns.load_artifact("baseline_v1.nsz")
candidate = ns.load_artifact("candidate_v2.nsz")

diff = ns.compare(baseline, candidate)

# Inspect structural alignment
print(f"Aligned steps:  {len(diff.aligned_steps)}")
print(f"Added steps:    {len(diff.added_steps)}")
print(f"Removed steps:  {len(diff.removed_steps)}")
print(f"Structural break: {diff.has_structural_break}")

# Inspect step alignment details
for aligned in diff.aligned_steps:
    print(f"  {aligned.step_name}: baseline={aligned.baseline_fp[:8]}  candidate={aligned.candidate_fp[:8]}")

Built-in Quality & Performance Metrics

The Regression Comparer Engine computes the following metrics out of the box:

Quality Metrics

• quality_delta — Evaluation score difference across standardized benchmark tasks. Positive means improvement; negative means regression.
• hallucination_delta — Change in hallucination score between pipeline versions. Positive means more hallucinated (worse); negative means more grounded (better).
• answer_faithfulness_delta — Change in how faithfully the system answers using retrieved context.
• retrieval_precision_delta — Change in the precision of retrieved documents. Degradation here often explains downstream quality regressions.
• answer_relevance_delta — Change in how relevant the answers are to the original queries.

Performance Metrics

• latency_step_breakdown — Step-by-step wall-clock latency diff. Pinpoints which API call, vector search, or model generation step became the new bottleneck.
• latency_p50_delta_ms / latency_p95_delta_ms — Median and tail latency changes.
• ttft_delta_ms — Change in Time-To-First-Token for LLM generation steps.
• throughput_delta — Change in tokens-per-second for generation steps.
• peak_vram_delta_mb — Change in peak GPU VRAM utilization.
• token_cost_delta — Total token usage and estimated cloud cost variance across runs.
• cosine_similarity — Cosine similarity over model activation hidden states at aligned layers. Low similarity at a specific layer often indicates model weight changes that affect reasoning.

Comparing Production Runs

import neuronscope as ns

baseline  = ns.load_artifact("rag_prod_v2.nsz")
candidate = ns.load_artifact("rag_prod_v3.nsz")

diff = ns.compare(baseline, candidate)

# Full summary
print(diff.summary())
# ┌─────────────────────────────┬──────────┬──────────┬──────────┐
# │ Metric                      │ Baseline │ Candidate│   Delta  │
# ├─────────────────────────────┼──────────┼──────────┼──────────┤
# │ Hallucination Score         │  0.127   │  0.089   │  -0.038  │
# │ Answer Faithfulness         │  0.841   │  0.907   │  +0.066  │
# │ Retrieval Precision         │  0.763   │  0.781   │  +0.018  │
# │ P95 Latency (ms)            │  1840    │  1620    │  -220ms  │
# │ Token Cost / Query          │  $0.0042 │  $0.0039 │  -$0.0003│
# └─────────────────────────────┴──────────┴──────────┴──────────┘

# Check for regression with configurable threshold
if diff.has_regression(threshold=0.05):
    print(f"REGRESSION DETECTED: {diff.regression_reason}")
    # "Hallucination score increased by 0.08 (threshold: 0.05)"
else:
    print(f"No regression. Quality improvement: {diff.quality_delta:+.3f}")

Comparing Two Models

When comparing two model versions on the same evaluation dataset, NeuronScope runs both models, captures telemetry from each, and performs a comprehensive cross-model analysis:

import neuronscope as ns

# Compare base model vs. fine-tuned model
base_model        = ns.Model.wrap(load_model("llama-2-"))
fine_tuned_model  = ns.Model.wrap(load_model("llama-2--finetuned"))
eval_dataset      = ns.Dataset.load("eval_queries.jsonl")

# Run evaluation on both models
comparison = ns.compare.models(
    baseline=base_model,
    candidate=fine_tuned_model,
    dataset=eval_dataset,
    metrics=["accuracy", "hallucination", "faithfulness", "latency", "cost"],
    sample_size=500,
)

print(f"Winner: {comparison.winner}")
print(f"Quality improvement: {comparison.quality_improvement:+.1%}")
print(f"Latency change:      {comparison.latency_delta_ms:+.0f}ms")
print(f"Cost change:         {comparison.cost_delta:+.4f} per query")

# Layer-level activation similarity (did the fine-tuning change internal representations?)
for layer_diff in comparison.layer_similarities:
    if layer_diff.cosine_similarity < 0.95:
        print(f"  Layer {layer_diff.name}: similarity={layer_diff.cosine_similarity:.3f} (changed significantly)")

RAG Pipeline Comparison

RAG pipeline comparisons are especially useful when changing retrieval strategies, re-indexing vector stores, updating embedding models, or modifying prompt templates. NeuronScope traces each component independently so you can isolate whether a quality change came from retrieval, context assembly, or generation.

import neuronscope as ns

# Capture baseline RAG run (current production)
with ns.Scope("rag_baseline") as baseline_scope:
    for query in eval_queries:
        docs = old_vector_store.search(query, k=5)
        baseline_scope.log_retrieval(query=query, docs=docs)
        answer = llm.generate(build_prompt(query, docs))
baseline_scope.save("rag_baseline.nsz")

# Capture candidate RAG run (new re-indexed vector store)
with ns.Scope("rag_candidate") as candidate_scope:
    for query in eval_queries:
        docs = new_vector_store.search(query, k=5)   # same queries, new index
        candidate_scope.log_retrieval(query=query, docs=docs)
        answer = llm.generate(build_prompt(query, docs))
candidate_scope.save("rag_candidate.nsz")

# Compare — attribute changes to specific RAG components
diff = ns.compare(
    ns.load_artifact("rag_baseline.nsz"),
    ns.load_artifact("rag_candidate.nsz"),
)

# Component-level attribution
print(f"Retrieval precision Δ:  {diff.retrieval_precision_delta:+.}")
print(f"Context relevance Δ:    {diff.context_relevance_delta:+.3f}")
print(f"Generation quality Δ:   {diff.generation_quality_delta:+.3f}")
print(f"→ Root cause: {diff.primary_regression_component}")

CI/CD Deployment Gating

NeuronScope's comparison engine integrates directly into CI/CD pipelines to gate deployments on quality thresholds — preventing regressions from reaching production automatically.

import neuronscope as ns
import sys

# Load baseline (from artifact registry or S3)
baseline  = ns.load_artifact("s3://my-bucket/artifacts/baseline_v2.nsz")
candidate = ns.load_artifact("candidate_v3.nsz")

diff = ns.compare(baseline, candidate)

THRESHOLDS = {
    "hallucination": 0.03,   # max allowed increase in hallucination
    "quality":       -0.05,  # max allowed quality decrease
    "latency_p95":   200,    # max allowed P95 latency increase (ms)
    "cost":          0.001,  # max allowed cost increase per query
}

regressions = []
if diff.hallucination_delta > THRESHOLDS["hallucination"]:
    regressions.append(f"Hallucination increased by {diff.hallucination_delta:+.}")
if diff.quality_delta < THRESHOLDS["quality"]:
    regressions.append(f"Quality decreased by {diff.quality_delta:+.}")
if diff.latency_p95_delta_ms > THRESHOLDS["latency_p95"]:
    regressions.append(f"P95 latency increased by {diff.latency_p95_delta_ms:+.}ms")

if regressions:
    print("DEPLOYMENT BLOCKED — Regressions detected:")
    for r in regressions:
        print(f"  ✗ {r}")
    sys.exit(1)

print("DEPLOYMENT APPROVED — No regressions detected.")
print(f"  Quality Δ:    {diff.quality_delta:+.3f}")
print(f"  Latency Δ:    {diff.latency_p95_delta_ms:+.0f}ms")

Interpreting System Deltas

A positive delta in the quality metric means the candidate is better. A positive delta in hallucination score means the candidate hallucinates more (worse). Here is a quick reference:

• Quality delta > 0 → Improvement. Candidate answers are more accurate.
• Hallucination delta > 0 → Regression. Candidate produces more ungrounded content.
• Latency delta > 0 → Regression. Candidate is slower.
• Cost delta > 0 → Regression. Candidate consumes more tokens or uses more expensive APIs.
• Retrieval precision delta > 0 → Improvement. The new vector store or embedding model is finding more relevant documents.

Structural vs. Numerical Changes

The Comparison Engine distinguishes between two types of changes:

Structural Changes

Structural changes affect the shape or composition of the AI system itself:

• A model layer was added or removed (architecture change).
• A retrieval step was added to or removed from the RAG pipeline.
• An agent tool was registered or deregistered.
• A workflow node was reordered or split.

Structural changes are detected by comparing architecture fingerprints. When a structural break is detected, the Comparer flags it with diff.has_structural_break = True and provides a structural diff summary.

Numerical Changes

Numerical changes affect the behavior of existing components without altering the system structure:

• Model fine-tuning (same architecture, different weights).
• Prompt template revision (same slots, different wording).
• Vector store re-indexing (same index structure, different vectors).
• Hyperparameter changes (temperature, top-p, max_tokens).

Numerical changes are detected by comparing weight fingerprints and evaluation metrics while the architecture fingerprint remains constant.