Deploy a cluster

A single Helix node is a complete system. A cluster exists so that losing one machine does not stop dispatch, and so a large operation can put a node where its people are. This page is the deployment: three nodes, the ports between them, and the one address your operators use.

The shape

Three nodes, each running the whole stack — its own MongoDB (a member of one replica set), the dispatch service, the SPA host, and, on exactly one of them, the optimiser:

Node	Started as	What it is
1	`create`, then `start replicant`	the writer at the start: it runs crud, and Spiral with it
2	`start replicant`	a voting partner; it takes over the writing if node 1 goes
3	`start standby`	votes for quorum, never elected — a regional viewer, reporting offload, a durable copy

Nodes 1 and 2 are the same kind of node. The only difference is that one of them created the cluster and the other joined it, and neither of those is a property the node keeps. Whichever holds the write is the primary, and that is decided by an election rather than by how it was started — so primary is something a card reports, never something you type.

Three votes, in three failure domains. Mongo elects on a majority of the votes it is configured with, so two nodes give you a second copy of the data but no failover: lose one and the survivor holds one vote of two, is never elected, and the cluster is dead rather than degraded. The third vote only helps if it is somewhere else — a standby sharing a host or a rack with the writer dies with it.

Ports

Each node publishes three, and only two of them concern the cluster:

Port	Published as	Who reaches it
MongoDB	the same number inside and out (`HELIX_MONGO_PORT`; chosen at start when unset)	the other members, in both directions — heartbeats are mutual
CRUD	`HELIX_PUBLIC_CRUD_PORT` when you set it, else a port chosen at start	the other nodes, forwarding writes to whichever node is writing
SPA	`HELIX_PORT`	your operators — and nothing else

Set the CRUD port explicitly. Left unset it is chosen afresh on every start, so the rule between nodes has to cover a whole range and moves when a node restarts.

Set the MongoDB port explicitly too, for a cluster. Left unset it is chosen from the high range on the first start of that node and then remembered in the node’s own helix.env, so it is stable across restarts — which is what the replica set requires, since a member’s address is host:port and is held in the set config. That is enough for a developer, but not for a firewalled estate: a chosen port means the rule between machines has to cover the whole range. Name it and the rule is one line.

The MongoDB port is published unchanged rather than remapped — the same number inside the container and out — because the number in the set config is the number reached from outside. It is per node, not one number for everyone: each member registers its own address, so a set may mix them. Two clusters on one machine each get their own and coexist; what you cannot do is run two nodes on the same port, so one node per host per port. Note that wiping a node’s storage discards the remembered port, so it comes back at an address the set was never told — a storage wipe and a rejoin belong together.

Each node also needs to know where it is:

export HELIX_PUBLIC_HOST=node1.example.com   # this node, as the others reach it
export HELIX_PORT=8001
export HELIX_MONGO_PORT=3001
export HELIX_PUBLIC_CRUD_PORT=8090
export HELIX_PRIMARY=node1.example.com:3001  # the joiners only
export HELIX_CLUSTER=acme-prod               # the cluster's name — set once, when you create it
export HELIX_SITE=london                     # where this node is (see "Locality" below)

Give HELIX_PUBLIC_HOST a name, not an IP address, wherever you can. The replica set records each member by that string, and mongod must recognise the string as itself before it will take the write. A name is aliased to loopback inside the container, so that check needs no network at all. An IP is not, so the node has to reach itself through its own published mongo port — and on a rootless runtime that port is forwarded in user space, which can accept the connection and drop it. The symptom is a start that waits and then reports no primary; the launcher warns at start when it sees an IP on a rootless runtime, and the fix is the host’s name ($(hostname), or its DNS name).

HELIX_CLUSTER names the cluster rather than the node. You give it once, on the node you create, and it is then kept in the world itself — so every member that joins carries it without being told again, and it is what your containers are called, which is how podman ps on a machine hosting two clusters tells you which is which.

HELIX_SITE is your own word for a location — a city, a building, a cloud region. It is what makes a node prefer its neighbours for reads, and what a client is told when it asks where it should be. Leave it unset if every node is in one machine room; there is no locality to express and nothing is lost.

Then helixctl <CLIENT> create on the first, and helixctl <CLIENT> start replicant / start standby on the others.

Where each node keeps what it holds

Install with --home <cluster> and a node lands in its own directory under ~/HelixRuntime/, with the cluster’s name beside it as a symlink:

~/HelixRuntime/
├── release/                     the core archive this machine was given — SHARED between instances
├── 7f3a9c21/                    this node; the directory never moves
│   ├── helixctl  VERSION  REGISTRY  manifest.json
│   ├── client/  client.tar  helix-config/
│   └── storage/
│       ├── data/                the database
│       ├── mongo/               mongod's own log
│       ├── logs/                crud.log, spiral.log, 2crud.log … rotated at each start
│       └── helix.env            this node's environment and secrets (mode 0600)
└── aa-prod -> 7f3a9c21/         the label — the only thing a rename touches

The three sub-paths are separately mountable, which is the point of them being separate: the database is fsync-heavy and random, the application logs are high-volume sequential writes, and on one filesystem the logs filling up stops the database. HELIX_DATA, HELIX_MONGO_LOG_DIR and HELIX_LOG_DIR_HOST override them individually, and moving one is an ordinary mount that Helix needs telling nothing about. Each node’s boot line says which device each path is on, and says so when two of them share one.

The directory is named by a stem, not by the cluster, because mount points live under it — a rename could not mv it — and because a name given up and later reused would point a new cluster at an old one’s database. So podman ps shows you the cluster’s name and df shows you the stem, and both are right.

Full detail, including what to back up: Storage and logs.

Signing in for the first time

create prints a line you need:

  administrator admin / 4913   (must be changed at first sign-in; not recorded anywhere)

That is the account and the code to sign in with. Helix invents it because a node that is running for real must not be claimable: on a production node an unknown name at the login screen is refused, so somebody has to be the administrator before anyone can do anything, and leaving that to a step you might forget is how a system ends up belonging to whoever opened it first.

The code is four digits, so it is deliberately temporary. Signing in with it gets you exactly one screen — choose your own password — and nothing else works until you have. Five wrong attempts lock the account until the node is restarted.

That line is the only time Helix says it. The code is not written to helix.env or anywhere else, because it is spent the moment the node first boots: an administrator is only ever seeded into an empty system, so after that first start the code could not be applied again even if something still held it — and a password kept in a file after it has stopped meaning anything is a liability, not a convenience. Note it down before you close the terminal. If you lose it before you have signed in once, the node is new and nothing is in it: empty storage/data (or remove the volume, if HELIX_DATA names one) and create again.

If you would rather choose the account yourself, set HELIX_SEED_USER and HELIX_SEED_PASSWORD before create and Helix uses those instead, with no forced change — it is your password, not one we invented. Setting the password to nothing (HELIX_SEED_PASSWORD=) says this administrator needs no password, which is what an unattended feed or a supervised node usually wants.

The joining nodes need none of this. They share the cluster’s database, and its administrator with it.

If every administrator password is lost

The cluster itself is never at risk from this. Nothing about starting a node consults a password — the world is whatever is in storage/data, and helixctl <CLIENT> start replicant brings it back whether or not anybody can sign in. What you have lost is the way in, and the way back is the database.

On the machine running the current writer, with the node up:

mongosh --port <the cluster's mongo port> helix \
  --eval 'db.user.updateOne({name:"admin"},{$unset:{passwordHash:1}})'

Then open the SPA, enter that name, and the login screen offers Set your password. You are back.

Four things worth knowing before you need this:

It must be the writer. A replica set takes writes only on its primary; the same command against a secondary is refused. helixctl <CLIENT> facts names the node, or Mission Control shows it.
mongosh is not in the Helix image — it is 187 MB for a handful of commands — so run it from the host against the published mongo port.
Setting HELIX_SEED_USER and HELIX_SEED_PASSWORD will not do it. Seeding applies only to an empty user table, deliberately: an environment variable that could take over a running system would be a worse problem than the one it solved.
This works because the replica set runs without authentication, which is also the reason its port must never be reachable from outside the cluster. Anyone who can run that command can become your administrator; that is the same set of people who can read your data files, but it is worth being deliberate about who they are.

You only see this once. The account is seeded into an empty system, so it is not a way to reset a password you have lost or to bring back an account somebody deleted — for that, another administrator uses the Admin screen. If nobody can sign in at all, the node’s crud.log says what it was missing.

Creating a cluster is not starting one. create makes a new cluster and refuses if this machine already holds a world — you only ever run it once, and it is the only command that asks for a name. From then on node 1 comes up with helixctl <CLIENT> start replicant, exactly like every other member.

A replicant works out which of the two acts it is doing from what it finds: told a HELIX_PRIMARY it joins that set, told nothing it brings back the world already on this machine, and given neither a world nor a primary it refuses rather than quietly creating a second cluster. That refusal is the point — master used to restart a world where it found one and silently make a new one where it did not.

Secrets — the Spiral licence, the maps key, your own client keys — are taken from the launching environment, so a service or cron wrapper must export them itself; a non-interactive shell does not read your profile. Every node needs its own Spiral licence, because the node that takes over runs the optimiser under its own name.

export HELIX_JWT_SECRET=<the same long random string on all three>

Sessions are signed tokens held in a cookie. A node that does not have the same key rejects a session another node issued — and that bites even without a load balancer, because administration, operations and every write are forwarded to the writer, on another host. Leave it unset and each node invents its own, which is right for a single machine and wrong here.

create mints it for you, and you only have to move it by hand if you commission by hand. The node you create generates one and records it in its own helix.env; a node commissioned from Mission Control is handed it as part of the environment it is given, so there is nothing to paste. Setting it yourself still wins, which is what you want when an existing estate already has a key.

It travels over ssh, never through the database. That is deliberate, and the reason is backups rather than joiners: a replicant replicates every password hash, so it is not a trust boundary — but the world is dumped to the vault, and tuning mounts a vault read-only into another node. A session key kept in the world would make “can read the vault” mean “can forge sessions on the live cluster”. So it is written to helix.env at mode 600 on the far machine, and the reply that comes back names the variables without their values.

How a client reaches the cluster

Any node will do, and that is the point. A client attaches to one node and takes its whole view from that node — the initial picture and every update after it. Writes go the other way: whichever node the client is talking to forwards them to the node that is currently writing, and it learns of a change of writer within about three seconds. So a browser never has to know which node is primary, and nothing at the client end changes when the primary moves.

That is already enough to run a cluster with no load balancer at all. What it does not do is tell a client which node it would be better off on, and in a cluster spread across sites that is the difference between a local hop and a trip over the link.

Locality: the cluster tells a client where to go

Every node answers GET /api/bind with a recommendation and the full member list:

{ "preferred": "https://leeds1.example.com:8001",
  "why": "network",
  "members": [ { "url": "…", "site": "london", "clients": 12 },
               { "url": "…", "site": "leeds",  "clients": 3 } ] }

The operator’s browser does this for itself: it signs in, asks, measures the members, uses the best of them and reports back what it measured. Anything else — a script, a host integration — can use preferred as given, and fall back through members if a node stops answering. See Access from outside the SPA.

How the recommendation is worked out, in order:

Rule	Basis
`learned`	a browser on the caller’s own network measured the nodes and reported the winner
`network`	the caller’s address is in the same IP network as a node’s
`answering`	nothing is known about the caller, so it is offered the site it reached
`load`	no locality configured; the node with the fewest clients

network is what makes this work on day one with nothing configured — a client on the same network as a node almost certainly belongs on it. learned is what makes it better than that over time, and it is the reason a curl script benefits from measurements it never took: the browsers do the measuring, the cluster remembers it per caller network, and everyone from that network is advised from it.

Nodes do not measure outward to clients, and cannot: a client behind NAT answers neither a ping nor an inbound connection. The measurement can only come from the client’s end, which is why it is collected there and shared through the cluster.

To pin a region’s clients to a region’s nodes whatever the measurements say, set HELIX_BIND_ALLOW on a node to the hosts it may offer. It is applied at the server, so a client cannot opt out of it.

One setting to let a client move between nodes

export HELIX_COOKIE_DOMAIN=.helix.example.com

A session is a cookie, and without this it belongs only to the node that issued it — so a client sent to a nearer node would have to sign in again on arrival. Setting the parent domain your nodes share lets the session travel with them.

Make it a domain of the cluster’s own — .helix.example.com, not .example.com. Every host under the domain you name here receives the session cookie, so a broad corporate domain would hand it to every unrelated application on it.

If your nodes do not share a parent domain, leave this unset. Clients stay on the node they reached, which works exactly as it did before; they simply do not get the benefit of moving.

Do you need a load balancer?

No. A client can be handed a list of nodes, ask any of them where it should be, and work down the list if one does not answer — which is how a database driver has always found a cluster. A VIP (keepalived, or your cloud’s floating address) is still useful as one address to hand out, and as a way to keep node names out of your configuration, but it is a convenience rather than a requirement, and it is not load balancing.

If you want one anyway — to terminate TLS in one place, or because your estate expects it — it must:

Pass WebSockets through. The operator’s live view is a WebSocket on the same port as the page. Allow Upgrade and do not buffer it.
Health check /api/ping, expecting 204. Never check / — the SPA answers 200 even when the backend behind it is dead.
Idle timeout of at least two minutes. The client heartbeats every 30 seconds and the server drops a silent connection after 60; a shorter idle timeout cuts healthy sessions.
Terminate TLS on 443 and set X-Forwarded-For and X-Forwarded-Proto yourself, overwriting anything the client sent. X-Forwarded-For matters more than it looks: it is how a node knows which network a caller is on, and therefore which node to recommend.
Leave the other two ports alone. MongoDB and the CRUD port are node-to-node traffic and must not be routed through the balancer.

frontend helix
    bind :443 ssl crt /etc/ssl/helix.pem
    http-request set-header X-Forwarded-Proto https
    default_backend helix_nodes

backend helix_nodes
    balance leastconn
    option httpchk GET /api/ping
    http-check expect status 204
    timeout tunnel 1h            # the operator's live view is long-lived
    server node1 node1.example.com:8001 check
    server node2 node2.example.com:8001 check
    server node3 node3.example.com:8001 check

Note that a balancer and the binding service pull in opposite directions: the balancer spreads clients evenly, and binding sends them where they are nearest. If you run both, the balancer decides where a client arrives and binding decides where it stays.

What an operator sees when the writer changes

The page stays up and the view keeps flowing, because it is being served by the node the browser is attached to. Writes pause for the election — a few seconds — and resume against the new writer with nothing to do at the client end.

A node stopped deliberately tells its clients so, and they move to another node straight away rather than waiting for it to come back.