NeuronScope
ArchitectureStablesince v0.6.0

Serialization flow

The bytes that end up in .nsz — how the header, manifest, chunk index, and columnar frames are laid out on disk.

Audience: ContributorsRead: 7 minEdit on GitHub Source

File layout

An .nsz file is a self-describing container. The header is fixed-size and lives at offset zero; every other section is located by the chunk index in the footer.

+----------------------+  offset 0
| header (128 bytes)   |  magic, format version, canonicalizer versions
+----------------------+
| manifest (JSON)      |  input fingerprints, probe schemas, tool version
+----------------------+
| column frames        |  zstd-compressed, dictionary-encoded categoricals
| ...                  |
+----------------------+
| chunk index          |  (column_id, frame_id) -> (offset, length, dtype)
+----------------------+
| footer (32 bytes)    |  index offset, index length, checksum
+----------------------+  EOF

Forward compatibility

  • Header and footer sizes are frozen. A change is a major version bump.
  • New column kinds are additive: they get a new type id and old readers ignore them.
  • Manifest is JSON precisely so third-party tooling can inspect it without a parser.

Atomic writes

The serializer writes to <target>.nsz.tmp, fsyncs it, and renames on top of the target path. A reader either sees the previous artifact or the new one — never a half-written file.

Related