NeuronScope
RecipeStablesince v0.7.0

Compare two models

A worked recipe: fingerprint two checkpoints, record them on the same input, and interpret the layer-wise diff.

Audience: Users comparing checkpointsRead: 8 minEdit on GitHub

You have two checkpoints of the same architecture and you want to know, precisely, where they differ. This recipe records both on the same input, aligns the recordings, and produces a layer-wise diff you can read.

Problem

Two BERT checkpoints — base and base-finetuned — give different answers on the same prompt. You want to attribute the disagreement to specific layers.

Prerequisites

  • NeuronScope v0.9.2 installed with the [torch,hf] extras.
  • Both checkpoints reachable from Hugging Face or a local path.
  • A single input the two models would legally accept.

Recipe

compare_models.py·
·python
import neuronscope as ns
from transformers import AutoModel, AutoTokenizer

BASE = "google-bert/bert-base-uncased"
TUNED = "my-org/bert-base-uncased-finetuned"

tok = AutoTokenizer.from_pretrained(BASE)
prompt = ns.Prompt("The quick brown fox jumps over the lazy dog.")
inputs = tok(prompt.text, return_tensors="pt")

# Record both on the same input with the same probe.
probe = ns.probes.LayerActivations(layers="encoder.layer.*")

with ns.record(model=AutoModel.from_pretrained(BASE), probe=probe) as a:
    a.forward(**inputs)
with ns.record(model=AutoModel.from_pretrained(TUNED), probe=probe) as b:
    b.forward(**inputs)

# Fingerprint check — same architecture, same input.
ns.assert_matches(a.model, b.model, kind="architecture")
ns.assert_matches(a.prompt, b.prompt, kind="prompt")

# Structured diff.
diff = ns.compare(a.artifact, b.artifact, metric="cosine")
diff.summary().to_markdown("compare.md")

Expected output

shellbash
$ cat compare.md | head -20
$ 
# Comparison — base.nsz vs tuned.nsz

| layer                     | cosine | l2      | verdict |
| ------------------------- | ------ | ------- | ------- |
| encoder.layer.0.output    | 0.998  | 0.031   | equal   |
| encoder.layer.1.output    | 0.994  | 0.048   | equal   |
| encoder.layer.2.output    | 0.981  | 0.126   | drift   |
| encoder.layer.3.output    | 0.942  | 0.310   | drift   |
| encoder.layer.4.output    | 0.812  | 0.892   | large   |
| encoder.layer.5.output    | 0.734  | 1.204   | large   |
| encoder.layer.6.output    | 0.681  | 1.517   | large   |
| ...

Interpreting the diff

The table above is the shape you want: near-equal representations in the early layers, then a clean divergence starting at layer 4. That is the signature of fine-tuning that reused the embedding and lower encoder stack and rewrote the higher layers.

Variants

  1. Batch input. Pass a list of prompts and get one diff per prompt. Aggregate with diff.by_prompt().
  2. Sub-selecting layers. Restrict the probe with a glob:layers="encoder.layer.[4-9].*".
  3. Cross-checkpoint CI. Save the reference .nsz in your repo and compare against it in CI — see the CI/CD recipe.
Related