Storage and logs

A Helix node holds four things on disk: a database, three kinds of log, a small environment file with its secrets in it, and the client bundle it serves. This page says where each one goes, how to move the heavy ones onto their own devices, and what to copy if you want to be able to get the node back.

The short version: everything a node holds lives under one storage/ directory, and every part of it is separately overridable. If you never set anything, you get a working layout in a predictable place. If you set the three variables below, you can put the database, the mongo log and the application logs on three different filesystems without telling Helix anything else.

The three ways Helix runs

Helix runs the same code three ways, and the only real difference is where the root is.

Deployed — ~/HelixRuntime/

This is a production or test node: no source, no build tools, just the launcher and the image.

~/HelixRuntime/
├── release/                     what this machine HOLDS — cores and bundles it can start on, SHARED
├── staging/                     what devops OFFERS this account → usually a symlink to an earlier
│                               environment's release/. Read-only in intent, and never pruned
├── 7f3a9c21/                    one instance — this directory never moves
│   ├── helixctl                 the launcher, taken out of the image
│   ├── VERSION  REGISTRY        the pin: which core, from where
│   ├── manifest.json            how this node came to be, and the command that restarts it
│   ├── client/  client.tar      the deployed bundle
│   ├── helix-config/            the client's domain, map and values
│   └── storage/
│       ├── data/                the database          (mongod --dbpath)
│       ├── mongo/               mongod's own log      (mongod --logpath)
│       ├── logs/                the application logs
│       └── helix.env            this node's environment, mode 0600
└── aa-prod -> 7f3a9c21/         the cluster's name, as a symlink

You get this by naming the cluster when you install:

tar xf helixcore-release-0.0.14.tar
cd helixcore-release-0.0.14
./helixctl install --home aa-prod        # -> ~/HelixRuntime/<stem>/, labelled aa-prod
cd ~/HelixRuntime/aa-prod
./helixctl import <client-bundle>.tar     # into release/; a start then selects it

release/ is beside the instances rather than inside one, so two nodes on one machine share a single copy of the core archive instead of keeping 650 MB each.

staging/ is how a newer version reaches the machine, and it is the one directory Helix only ever reads. Devops points it at an earlier environment’s release/ — usually a symbolic link — so the artefact names and shapes match by construction. helixctl import copies out of it into release/; nothing is ever written back, and it is never swept by the prune, because it belongs to another environment. That direction is deliberate: a release is pulled by the account that wants it, never pushed into a test or production account by whoever built it, so no credential has to exist anywhere for this to work.

Image + repo — a client repository

This is development, and a repository is the root. helix/runtime/ is HELIX_HOME.

<client-repo>/
├── helix/
│   ├── custom/                  the client extension source        ← tracked
│   ├── scenarios/               scenario data                      ← tracked
│   └── runtime/
│       ├── helixctl             a committed bootstrap seed         ← tracked
│       ├── VERSION  SDK_VERSION  CLIENT_REGISTRY   the pin         ← tracked
│       ├── client/  client.tar  helix-config/      build output    ← ignored
│       ├── manifest.json        a fact about THIS machine          ← ignored
│       └── storage/             everything this node holds         ← ignored
└── tests/                       run files tracked, the harness pulled

release/ is a sibling of runtime/ here, not a shared directory, because in a repository it is build output rather than something the machine was handed.

Bare metal — no container at all

startHelix <CLIENT> dev runs mongod, crud and the SPA directly. It keeps its state under the repository it found, or under your home directory when there is no repository — and one directory per session, named for the launching process, whatever the start type:

<repo>/helix/runtime/storage/bare/session-<pid>/{data,mongo,logs}   any bare-metal session
~/helix/<CLIENT>/storage/bare/session-<pid>/...                     when no client repo is in play

The launcher prints the path on its one [startHelix] boot line (logs:<path>), which is where to read it from. A start sweeps the sibling directories whose pid has gone and leaves the rest alone: it used to be one directory per mode, which meant a mode could not run twice at once and each new start rotated the previous session’s history away.

A bare-metal run is disposable and says so: a session gets an empty data/, so each run begins with an empty database. session-<pid> is per process, so several can run at once without colliding — and none of them shares a directory with a container, which is the collision this layout exists to prevent.

All three at once, in one account

Nothing in the above overlaps, deliberately: a deployed node is under ~/HelixRuntime/<stem>/, a repository node is under its own checkout, and bare metal is under storage/bare/<session>/. You can run all three simultaneously in one login and none of them touches another’s files.

Why it is one directory

storage/ is a single directory for one reason, and it is worth being blunt about it: a secrets file was once committed to a client’s own repository. The repository ignored runtime artefacts by naming them one at a time — client/, client.tar, helix-config/, logs/, manifest.json — and when helix.env arrived later, nobody added it to the list.

A list of things to ignore is a list that has to be maintained. One directory is a rule that cannot be escaped:

/helix/runtime/storage/

So the division is: what a client writes is tracked, what a node holds is not. The pin is tracked (VERSION, SDK_VERSION) because it is a decision someone made; the bundle is not, because it is built. The database, the logs and helix.env are not, because they belong to one machine.

helix.env is mode 0600 and it is not a config file you are expected to edit — though you may. helixctl writes it at the first launch and merges into it afterwards: every line already there is kept byte for byte and only names it does not yet carry are added. It holds the cluster’s identity (its name, its uuid, the session key every member shares) alongside the secrets, which is why a restart does not need to be told any of it again.

Three mount points, for three kinds of I/O

The three sub-paths are separately overridable because they are three genuinely different workloads, and on a busy node they compete:

Variable	Default	Becomes	Workload
HELIX_DATA	storage/data	mongod --dbpath	fsync-heavy, random
HELIX_MONGO_LOG_DIR	storage/mongo (cluster only)	mongod --logpath	sequential, low volume
HELIX_LOG_DIR_HOST	storage/logs (cluster only)	the application logs	sequential, high volume

The two log defaults apply to the cluster modes. Every other mode — dev, tuning, demo, spiral, spiralOnly — mounts no log directory unless you name one, because none of them keeps a world either. See “Logs in the container” below.

The application logs are the surprising one. The two highest-volume writers in the system are the tees between crud and Spiral, and they are not level-gated — on a busy node they can out-write the database. On one spindle they contend with it; on one filesystem, the logs filling up stops the database.

Moving one is an ordinary mount. Nothing in Helix needs to know:

# a dedicated device for the database
sudo mkfs.ext4 /dev/nvme1n1
mkdir -p ~/HelixRuntime/aa-prod/storage/data
sudo mount /dev/nvme1n1 ~/HelixRuntime/aa-prod/storage/data
sudo chown $(id -u):$(id -g) ~/HelixRuntime/aa-prod/storage/data

and permanently, in /etc/fstab — note the real path, not the label (see below):

/dev/nvme1n1  /home/helix/HelixRuntime/7f3a9c21/storage/data  ext4  defaults,noatime  0 2
/dev/nvme2n1  /home/helix/HelixRuntime/7f3a9c21/storage/logs  ext4  defaults,noatime  0 2

Mount before the first start, and own the directory yourself. A freshly mounted filesystem is owned by root; a rootless container maps your uid into itself, so mongod is denied on a directory it cannot write. The chown above is the whole of the fix.

The node tells you whether it worked. Its boot line names the backing of each path, and says so when two of them share a device:

[startHelix] start=create mode=replicant env=live joinable=1 client=AA core=0.0.15 … |
  mongo:durable(ext4)@/data/db logs:/app/logs mongo-log:/app/mongo client-src:tar:… |
  app:3000 ws:3000 crud:8090 mongoPort:3001  [logs and data share /dev/nvme0n1p2 — they contend, and either can fill it]

That clause is the answer to “did my mount take”. If you moved the logs and it is still there, something is mounted somewhere other than you think.

Setting them explicitly

export HELIX_DATA=/srv/helix-data                # a path, or a container volume NAME
export HELIX_MONGO_LOG_DIR=/var/log/helix/mongo
export HELIX_LOG_DIR_HOST=/var/log/helix/app

HELIX_DATA accepts either a path or a container volume name, and it tells them apart the way the container runtimes do: anything with a slash in it is a path. A bare word is a volume, which is how existing development boxes keep working — helix-<CLIENT>-data is still found and still used.

A value that is meant to be a directory and isn’t one is refused with a message rather than guessed at. This matters more than it sounds: HELIX_LOG_DIR_HOST=1 once became a container volume literally named 1, and cost an afternoon.

Logs: what rotates, and when

The application logs rotate at every node start, keeping three closed generations:

crud.log    crud.log.1    crud.log.2    crud.log.3

crud.log.3 is dropped, .2 becomes .3, and so on. HELIX_LOG_KEEP changes how many are kept; the default is 3. The files that rotate:

File	What it is
crud.log	what the node reports about itself — errors only, by default; see below
2crud.log	every crud command the node was given, for replay and debugging
crud2spiral.log	what crud sent Spiral
spiral.log	what Spiral said back
serve.log	the SPA host
vehicle.log	the vehicle feed, where there is one
mongo.log	mongod’s own log, in storage/mongo/

An empty crud.log is the healthy case

A container node runs at LOG_LEVEL=ERROR, so crud.log holds nothing at all unless something went wrong. That is deliberate — a stray LOG_LEVEL=DEBUG in a launching shell once flooded a container’s log to gigabytes and stalled the node, so the ambient variable is ignored entirely and the default is off. Only dev and tuning, which mount a live repository, default to INFO.

So do not read an empty crud.log as a node that is not running. To make it an account of the run rather than only of its failures:

HELIX_LOG_LEVEL=INFO helixctl <CLIENT> start replicant     # or PERF for milestones only, DEBUG for everything

PERF is the useful middle: errors plus the timing milestones, without the per-second inflate traffic that INFO carries. The schema gate’s verdict — which world this build found and what it did about it — is an INFO line, so it is one of the things you only see when you ask.

dispatch.log is not in that list, and there is no such file in EITHER world. dispatch does not redirect its own output: in a container it is the container’s own log, read with podman logs (or docker logs), and bare metal it is the terminal you launched from — startHelix backgrounds dispatch and inherits its streams rather than capturing them. Bare metal used to write a file, which is why other pages have promised one; a sample HELIX_LOG_DIR_HOST that lists dispatch.log is promising something that will not be there. If dispatch’s output needs keeping, redirect the launch.

A node start is the boundary, not a restart of something inside it. That distinction is deliberate in two places. Spiral can be relaunched inside a running node — a planned restart keeps the node rolling — and when it is, spiral.log and crud2spiral.log are appended to rather than rotated, so one node’s session reads as one file. crud can also be relaunched inside a node, which a replicant promoted to writer does; crud.log and 2crud.log are truncated when that happens, so what rotation preserves is the generations from previous node starts.

What mongod does and does not do for itself

Very little, and one piece of it is a trap:

• mongod over-writes its log at every start unless told otherwise. --logappend is passed, so it does not: the log a crash left behind is still there after the restart that followed it.
• mongod has no rotation. There is no SIGUSR1 handler and nothing issues a logRotate command, so mongo.log is rotated by Helix at node start with everything else — before mongod --fork, because renaming a file under a running mongod leaves it writing to the moved inode.

So mongo’s log behaves like every other log here, while remaining entirely mongod’s own business to write.

Logs in the container, and why you want them on the host

Everything above is written inside the container, and --rm takes it away when the run ends. For a run that finished cleanly that is correct — there is nothing to explain. For one that did not, the account of why went with it.

A cluster node therefore gets a host log mount by default (storage/logs), because a production node must be able to say why it stopped.

Every solo mode — dev, tuning, demo — is opt-in, and so are spiral and spiralOnly. Set HELIX_LOG_DIR_HOST and the logs land there; leave it and they stay in the container and go with it. Two reasons, and the second is the one that bites:

• A node that keeps no world has no business keeping a record of itself. These modes take everything of value from the client’s repo and write everything of value back to it — the domain, the map, run files, captures, the built bundle — and nothing else they produce is meant to outlive the run.
• storage/logs is one directory per node home, so every solo run from the same repo shared it. Two of them at once interleaved into one set of files, and each new start rotated the other’s history away. Opt-in means the operator names a directory per session, which is what the demo gateway already did.

This changed in 0.0.16. dev and tuning used to mount storage/logs whether you asked or not; demo was already opt-in. If a script of yours went looking in storage/logs after a dev run, give it HELIX_LOG_DIR_HOST.

The level, and the auxiliary logs, follow the same answer

HELIX_LOGS_EPHEMERAL sends mongo.log, serve.log and vehicle.log to /dev/null and suppresses 2crud.log, because logs nothing can read are pure cost. It is observed, not declared: startHelix asks findmnt whether anything is mounted at /app/logs and sets it from the answer. So it follows your choice rather than the mode’s name — which is the point, because the two stopped coinciding once every solo mode became opt-in.

The log level follows it too. A simulator whose logs are being kept runs at INFO, which is what makes dispatch’s [inflate]/[init] load timings visible; one whose logs are thrown away runs at ERROR. A cluster node stays at ERROR whatever its mounts, because it is a live node and not somebody’s bench. HELIX_LOG_LEVEL overrides all of it, in either direction.

(HELIX_LOGS_EPHEMERAL was HELIX_DEMO=1 until 0.0.15, when the deployment class it belonged to was retired — one boolean beside an enum that already had a value for the case — and was keyed on the demo start type until 0.0.16, which was the same mistake one level up: a declaration about a start type answering a question about a mount.)

The instance directory and its name

A deployed instance lives in a directory named by an 8-hex-digit stem — 7f3a9c21 — with the cluster’s name beside it as a symlink. That looks like indirection for its own sake until you try to rename a cluster:

• Mount points live under the instance directory. storage/data and storage/logs may be entries in /etc/fstab. A rename that had to mv the directory could not.
• A name can be given up and reused. If the directory were named after the cluster, a new cluster taking a retired name would be pointed at an old one’s database.

So the stem is minted once, means nothing, and never changes — which is exactly what a mount point needs. The name is a label, and renaming is one command:

ln -sfn 7f3a9c21 ~/HelixRuntime/acme-prod     # the new name
rm ~/HelixRuntime/aa-prod                     # the old one, when you are ready

Nothing else moves, no mount is disturbed, and a node that was running stays running. Every path Helix uses is resolved through the symlink to the real directory, so the label can change underneath a live node without aiming anything at a different instance.

Which means two tools show you two different things, and both are right:

podman ps --format '{{.Names}}'       # helix-aa-prod        — the NAME
df -h ~/HelixRuntime/aa-prod/storage  # …/HelixRuntime/7f3a9c21/storage — the PATH

The container is named after the cluster, because that is what tells you which of two clusters on a machine you are looking at. df, mount and /etc/fstab name the stem, because that is what does not move.

What to back up

Nothing on this list is obvious from the layout, so here it is plainly.

Path	Back it up?	Why
storage/data/	yes, or rely on the replica set	the world. In a cluster it is already on every member
storage/helix.env	yes, and keep it secret	the session key, the licence, the cluster’s uuid. Losing it does not lose the data, but every session is invalidated and a rejoining member has to be told everything again
storage/logs/, storage/mongo/	no	evidence, not state. They rotate away by design
client/, client.tar, helix-config/	no	build output — rebuild from the client’s repository at the recorded version
VERSION, REGISTRY, manifest.json	cheap, so yes	what this node runs and how it got there. Small, and it is what lets another machine reproduce it
~/HelixRuntime/release/	no	the core archive, re-fetchable — unless this machine is air-gapped, in which case it is the only copy and it is how you build the next node

In a cluster, the replica set is the backup for the data and nothing else. A third member in a third failure domain protects storage/data; it does not protect helix.env, and it is not a substitute for a copy taken at a point in time — a bad write replicates as faithfully as a good one.

If you have capture switched on, the vault is a fourth kind of state and the only one with its own retention policy. See Capture and extract.

Checking a machine is clean

The layout above only helps if the machine it lands on is tidy and its environment is right. This is a checklist rather than a script, because three of the six steps end in a decision rather than an action.

1. Which runtime is in use, and are the images in it?

helixctl facts | grep -E '"runtime|"image'

A machine with both podman and docker installed has two image stores, and helixctl picks one (podman when it is version 4 or newer; HELIX_CONTAINER_RUNTIME overrides). Images loaded into the other one are invisible to it, so every run re-fetches ~650 MB into the store it does use. facts names the runtime and says whether the image it would run is present.

2. Is the store healthy?

podman system df        # or: docker system df

An error here is the whole finding, and it can look like something else entirely. A single half-written image — a blob with no manifest, which is what an interrupted pull leaves — makes every listing fail:

Error: locating item named "manifest" for image with ID f3e8e2a… : file does not exist

podman images then reports nothing, which reads as an empty store rather than a broken one. The repair is the narrow one the error suggests, and it costs nothing else:

podman rmi --force <the id in the message>

Do not reach for podman system reset for this. It clears the store and destroys every volume with it, which is every world on the machine.

3. What is reclaimable

helixctl prune --dry-run      # old cores: keeps the newest 2 + pinned + in-use + the floats
podman image prune -f         # dangling layers nothing references

On a development box this is usually the largest single reclaim available, and most of it is not the cores — it is the untagged intermediate layers behind them.

prune covers the core repository only. A client’s own images (helixctl release --as image publishes to CLIENT_REGISTRY) are outside it and accumulate for ever. They cost far less than their nominal size because they share the core’s layers, but nothing will ever remove them, so a long-lived build box wants an occasional look:

podman images <your client registry>/<client>

4. Orphaned volumes — a decision, not a sweep

podman volume ls

Three kinds, and only one is safe to remove without thinking:

pattern	what it is	safe?
helix-<CLIENT>-modules	an npm cache for a client’s dev build	yes — it re-fills
helix-<CLIENT>-data	a world. A development node’s database	no — decide per volume
anything else helix-*-data	a world from a test run, or a retired convention	decide, then remove

Every *-data volume holds a database. podman volume rm on one is not tidying, it is deleting a world, and there is no confirmation step. Look at what a node still uses before removing anything:

helixctl <CLIENT> facts | grep -E '"dataVolume|"volumeExists'

5. Legacy paths

Things that moved and may have left something behind:

path	why it is there
~/helix/<CLIENT>/logs/	superseded — bare metal keeps its state in storage/bare/<session>/ now
<repo>/helix/runtime/logs/	superseded by storage/logs/
<repo>/helix/runtime/helix.env	superseded by storage/helix.env, and still honoured where it exists
a stray 2crud.log beside the source	the relative path it used to be opened with
empty directory trees	the fossil of a bind mount whose source did not exist; -v creates one silently

6. The environment — reachable without a profile?

A profile that exports SPIRAL_LICENCE, GOOGLE_MAPS_API_KEY, HELIX_JWT_SECRET and the rest is convenient for interactive work and invisible to everything else: a systemd unit, a cron job and a forced ssh command read no profile. So the check is not “are these set” but “are they set for the thing that will start the node”.

env -i HOME="$HOME" bash -lc 'helixctl <CLIENT> facts' | grep -E '"clientTar|"image'   # a bare environment

Where they belong instead:

• values.json in the client’s bundle, for anything the client declares — the licence tiers among them.
• storage/helix.env, for this machine’s own: written by the first launch, merged into afterwards, and sourced by every later one. That is what makes a node restartable by a service rather than only by a person.

Once both carry what a node needs, a profile export is a convenience rather than a second source of truth — and worth deciding which it is, because the failure mode of the second is a node that works when you start it and not when anything else does.

Moving a node’s whole root

Everything above assumes the defaults. One variable overrides the lot:

export HELIX_HOME=/opt/aa-config

HELIX_HOME is the node’s root, and every path in this page is derived from it — so a client whose own software distribution installs Helix somewhere of its own needs nothing else. HELIX_STORAGE_DIR moves just the storage/ directory, and HELIX_RUNTIME_ROOT moves ~/HelixRuntime itself, for a machine that keeps its instances somewhere like /srv/helix.

See also Deploy a cluster for the ports and node roles, and Versions and upgrades for what changes on disk when a node moves to a new core.