Large datasets

Set a memory budget

The recorder's ring buffer is the single knob for peak memory. Setting it too low causes back-pressure on the dataset iterator; setting it too high risks OOM under a warm cache.

import neuronscope as ns

with ns.Recorder(model, buffer_mb=1024, stall_timeout_s=60) as rec:
    rec.record(ds.chunks(rows_per_chunk=512))
    rec.save("run.nsz")

Chunk the dataset

Dataset.chunks(n) yields sub-iterators of n rows each. The fingerprint is preserved because chunking is deterministic in the original row order.

Sample instead of scan

• Never combine Dataset.shuffle with a scan — you lose the reproducibility guarantee for the same fingerprint.
• Prefer chunked recording followed by offline aggregation over one giant run.