Capture and extract

Something happened on a live system at half past two, and you want it as a test. This page is how: turn on data logging, let the vault accumulate beside the node, and later — on a machine that has no path into the live cluster at all — extract the world as it was just before a chosen command, with the commands that followed.

The result is an ordinary test. Open it in the Spiral Laboratory, step it, conclude it, minimise it, keep it.

The firewall between live and development is a disc, not an API

Nothing in this mechanism lets a development machine reach a live node. The live node writes to a directory. A tuning node reads that directory, mounted read-only, and does all its work locally. There is no network path, no port, no credential, and nothing on the live side that answers a request.

And what crosses is Spiral data only: the extract is a position and a command slice in the optimiser’s own vocabulary. Your business records stay in the live database. See what the laboratory holds.

1. Data logging

With data logging on, every line the node sends Spiral is also written to Mongo as data, and each tagged command — the ones a mutation carries — writes a marker the instant its processing begins.

Turn it on per instance in Mission Control (the Capture block on the writer’s card), or from the start with HELIX_DATA_LOGGING=1. It is on by default for dev and tuning, off for demo and the cluster modes until you ask for it.

The cost is one small write per command and roughly double the oplog volume while it is on. Measured on a 9,726-command stress load: the load inflated in 21 seconds with logging on, against 22 without.

The stream is kept for HELIX_STREAM_TTL_HOURS (168 by default) and then expires.

Why the marker is where it is

The marker is written at the start of a tagged command’s processing, before the entity write, inside a critical section that serialises the whole of it. That is what makes a cut exact: everything before the marker is the world as it was, and the command itself has not happened yet — precisely the state a node taking over the cluster would replay. So “the world before tag N” is one point in one totally ordered log, not a reconciliation of several.

2. The vault

The vault is a directory the node fills with everything an extract needs:

dumps/<when>/	a full database dump, taken when the world is reset and every HELIX_VAULT_DUMP_HOURS (24)
oplog/	slices of Mongo’s operation log, every HELIX_VAULT_SLICE_SECONDS (60)
markers.jsonl	the tag markers found in those slices — the index you pick from
stream/	the commands themselves, per Spiral instance

Give any node a vault by passing a host directory:

HELIX_VAULT_DIR_HOST=/srv/helix-vault  helixctl AA start standby     # the cluster's capture member
HELIX_VAULT_DIR_HOST=/tmp/vault        helixctl AA start dev          # one box, the whole round trip

On a cluster a standby member is where it belongs: a member at priority 0 that can never be elected and never becomes primary, so the work costs the live system nothing but disc. Mission Control shows what the vault holds and how far back it reaches.

Retention is HELIX_VAULT_KEEP_DUMPS (7) and HELIX_VAULT_KEEP_OPLOG_HOURS (168). A tag is extractable once its slice has been written, so the lag is at most one slice period.

The vault is a fourth kind of state, and the only one with a retention policy of its own. A node already holds three things under its storage/ directory — the database, mongod’s log and the application logs — and those are bounded by rotation or by not growing. The vault is bounded by the two variables above, and by nothing else: it grows with how long you keep it and how busy the node is. Two consequences worth planning for.

Give it its own directory, and preferably its own device. HELIX_VAULT_DIR_HOST is a host path with no default, deliberately — a vault is opt-in and it is the one part of a node that will fill a disc if nobody chose where it goes. A conventional place is beside the rest of a node’s state without being inside it:

HELIX_VAULT_DIR_HOST=~/HelixRuntime/aa-prod/vault  helixctl AA start standby

That keeps everything about the node under one root while leaving the vault outside storage/, so a mount, a backup rule or a sweep can treat it separately from the world it was taken from.

It is not a backup of the world, and it is not covered by one. A vault holds dumps and oplog slices for replay — reconstructing what a moment looked like so Spiral can be run against it again. Backing up storage/data does not back up the vault, and keeping the vault does not mean you can restore the cluster. See Storage and logs for what to copy and why.

One vault, one world

A vault belongs to one world. Each world carries a lineage marker that survives a restore, so a live set restarted from a dump keeps appending to the same vault. If a node finds a vault written for a different world, what happens depends on what the node is: dev, tuning and demo empty it and start again — a development box wipes its database at every start, and its operator should not have to think about it — and a production node refuses to touch it, reporting why in its heartbeat until an operator moves it aside or sets HELIX_VAULT_RESET=1 for one deliberate reset.

3. Reach the vault from a tuning node

Mount the same directory read-only. The transport is yours — the same host, a read-only NFS export, or a copy:

HELIX_VAULT_DIR_HOST=/srv/helix-vault:ro  helixctl AA start tuning

The :ro is what makes it a source: the node reads it and never runs the vault job.

4. Extract a moment

In the laboratory, Create a new test… → from vault. Pick the date, then the tag from the markers in that window: each row shows the instance, the tag, when it happened, whether it was a clean moment, and the first part of the command. A tag whose slice has not reached the vault yet says so.

The picker reads as the world before tag N, replaying from N. Choose how far to replay — to a tag, or a number of steps — and the node does the rest, locally: it restores the newest dump that precedes the marker onto a throwaway database, rolls the operation log forward to exactly that marker, builds the starting position from the restored world, appends the command slice, and writes the test into development/.

Static start is ticked by default. It writes the legacy pattern — gear 10, then staticStart — before the schedule, so Spiral takes the position exactly as given and does not optimise until the first replayed command. That is what makes the replay start from the plan the live system actually had.

If a stop in the position has no schedule of its own at the cut — Spiral had not answered for it yet — its schedule-owned fields are recovered from its own input line in the stream. If a stop still cannot be described, the extract refuses the tag and names the nearest clean ones rather than writing a test that Spiral would reject.

On an AA-scale world this takes about a minute and a half: a position of 8,000 entities restored, rolled forward and written.

The other two sources

The same menu offers two that need no vault:

• from capture — write the current position into a new test and record everything sent after it. On a live node, Snapshot in the Dispatch header does the same thing and becomes its own stop button. On a solo node it records into the laboratory instead (see Record your own steps).
• from logs / from stream — build a test from what this node has already sent Spiral, either from the log file or, with data logging on, from the stream sliced at a tag. This is the one to reach for after a crash: there is no live optimiser left to capture from, but the exact stream that killed it is still there, and the test replays standalone.

5. Check it says what the live system said

Open the extract in the laboratory and play it. The answer should match what the live world did at that moment — the same resource, the same desk, the same state. Proven on stress test 10: an extract at a mid-run tag reproduced the allocation of every stop in the slice exactly.

Then conclude it, minimise it, and promote it — see the Spiral Laboratory.

Settings

Variable	Default	What it does
HELIX_DATA_LOGGING	1 for dev/tuning, else 0	record the crud→Spiral stream and the tag markers
HELIX_STREAM_TTL_HOURS	168	how long the stream is kept in the database
HELIX_VAULT_DIR_HOST	—	the host directory to use as this node’s vault; :ro makes it a read-only source
HELIX_VAULT_SLICE_SECONDS	60	how often an operation-log slice is written
HELIX_VAULT_DUMP_HOURS	24	how often a full dump is taken
HELIX_VAULT_KEEP_DUMPS	7	dumps kept
HELIX_VAULT_KEEP_OPLOG_HOURS	168	operation-log slices kept
HELIX_VAULT_RESET	—	1 lets a production node reset a foreign vault, once
HELIX_CANDIDATES_DIR_HOST	—	a directory of Spiral builds the laboratory can test against

Related

• The Spiral Laboratory — what to do with the test once you have it
• Versions and upgrades — the node modes and how they are launched
• Develop a client extension — the dev and tuning environments