NeuronScope
ArchitectureStablesince v0.6.0

Telemetry Serialization Flow & Layout

Binary layout of .nsz telemetry container files — self-describing headers, input fingerprints, columnar activation streams, and chunk index mapping.

Audience: Storage Engineers & System ArchitectsRead: 8 minEdit on GitHub Source

The .nsz container format is engineered for high-throughput AI telemetry serialization. It lays out metadata, execution plans, and activation tensors into contiguous binary columns, enabling zero-copy range reads over network streams.

On-Disk Container Layout

+----------------------+  offset 0
| header (128 bytes)   |  magic, nsz format version, canonicalizer digests
+----------------------+
| manifest (JSON)      |  input fingerprints (model, prompt, dataset, env)
+----------------------+
| telemetry frames     |  columnar activations, tool calls, latency spans
| ...                  |  compressed with zstd binary encoding
+----------------------+
| chunk index          |  (column_id, step_id) -> (offset, length, dtype)
+----------------------+
| footer (32 bytes)    |  index offset, index length, BLAKE3 checksum
+----------------------+  EOF

Container Binary Sections

  • Header: Fixed 128-byte header storing format version and magic signature for instant validation.
  • Manifest: JSON document embedding canonical input fingerprints (fp:model, fp:prompt, fp:dataset, fp:env).
  • Telemetry Frames: Contiguous zstd-compressed binary columns storing layer activation matrices and execution spans.
  • Chunk Index & Footer: Offset table mapping layer paths to byte ranges, enabling O(1) partial range queries from cloud S3/GCS buckets.

Reader Compatibility Contract

Atomic File Writes & Cloud I/O

To prevent corrupt or partial telemetry files during unexpected process termination, serializers write to temporary spool files (.tmp.nsz), fsync disk buffers, and atomically swap into place upon session completion.

Related