ClusterHost
A deployment on several machines has one coordinator and one host agent per machine that runs workers:
- The coordinator is a
ClusterManagerwith theremoteoption. It keeps everything aClusterManagerdoes: the shard-to-cluster map, IDENTIFY slots, routing, health and recovery. It listens for hosts on a WebSocket. - A host runs
ClusterHost. It connects to the coordinator, starts its own worker script for each cluster placed on it, and relays the workersâ IPC frames both ways.
A cluster on another machine therefore behaves like a local one: the coordinator supervises it, routes ShardBus messages to it, and paces its IDENTIFYs together with every other cluster. The sharding guide has the overview.
coordinator (ClusterManager + remote) host "eu-1" (ClusterHost) ââ shard â cluster map, IDENTIFY slots ââwssââś ââ worker.ts cluster 0 ââ routing, health, restarts ââ worker.ts cluster 2 ââ listener on remote.port ââwssââś host "eu-2": cluster 1, cluster 3import { ClusterManager, ClusterHost, RemoteClose, REMOTE_PROTOCOL_VERSION } from "lunibee";import type { RemoteOptions, HostStatus, ClusterHostOptions, ClusterHostEvents } from "lunibee";// or the package on its own:import { ClusterManager, ClusterHost } from "@lunibee/sharding";import { ClusterManager, ClusterHost, RemoteClose, REMOTE_PROTOCOL_VERSION } from "lunibee";// or the package on its own:import { ClusterManager, ClusterHost } from "@lunibee/sharding";Quick start
Section titled âQuick startâimport { ClusterManager } from "lunibee";
const clusters = new ClusterManager({ token: process.env.DISCORD_TOKEN!, script: "./worker.ts", // only used for clusters on this machine (remote.local) shardCount: "auto", clusterCount: 6, waitForReady: true, remote: { port: 7070, hostname: "0.0.0.0", // the default, 127.0.0.1, is reachable from this machine only tls: { cert: process.env.TLS_CERT!, key: process.env.TLS_KEY! }, secret: process.env.CLUSTER_SECRET!, // 16+ characters, shared with every host hosts: ["eu-1", "eu-2", "us-1"], // clusters are placed on these, in this order },});
clusters.on("hostJoin", (id) => console.log(`host ${id} joined`));clusters.on("hostLeave", (id, reason) => console.warn(`host ${id} left: ${reason}`));
await clusters.spawn(); // waits for the hosts, then places clusters on them// host.ts, on each machineimport { ClusterHost } from "lunibee";
const host = new ClusterHost({ url: "wss://coordinator.example.com:7070", secret: process.env.CLUSTER_SECRET!, id: "eu-1", script: "./worker.ts", // the host decides what runs; the coordinator cannot});
host.on("disconnected", ({ code, reason }) => console.warn(`lost coordinator: ${code} ${reason}`));host.on("error", (error) => console.error(error));
await host.connect(); // rejects if the first attempt fails; reconnects by itself laterprocess.on("SIGTERM", () => void host.close());import { ClusterManager } from "lunibee";
const clusters = new ClusterManager({ token: process.env.DISCORD_TOKEN, script: "./worker.js", shardCount: "auto", clusterCount: 6, waitForReady: true, remote: { port: 7070, hostname: "0.0.0.0", tls: { cert: process.env.TLS_CERT, key: process.env.TLS_KEY }, secret: process.env.CLUSTER_SECRET, hosts: ["eu-1", "eu-2", "us-1"], },});
await clusters.spawn();// host.js, on each machineimport { ClusterHost } from "lunibee";
const host = new ClusterHost({ url: "wss://coordinator.example.com:7070", secret: process.env.CLUSTER_SECRET, id: "eu-1", script: "./worker.js",});
await host.connect();process.on("SIGTERM", () => void host.close());The worker script is the same as for local clusters (a ShardManager that reads the worker contract). The bot token is never sent to a host: each worker reads it from its hostâs own environment.
Coordinator options
Section titled âCoordinator optionsâremote on ClusterManagerOptions is a RemoteOptions. It is validated in the ClusterManager constructor; the listener opens in spawn().
| Option | Type | Default | Description |
|---|---|---|---|
port |
number |
â | TCP port. Required; integer from 0 to 65535. 0 picks a free port; read it from clusters.remotePort after spawn() started. |
hostname |
string |
"127.0.0.1" |
Interface to bind. The default is reachable from this machine only. Any non-loopback address needs tls or allowInsecure. |
secret |
string |
â | Shared secret, at least 16 characters. Required. Hosts prove they know it; it never crosses the wire. |
tls |
{ cert: string; key: string } |
none | Serve wss://. Both must be non-empty strings (PEM contents, as Bun.serve takes them). |
allowInsecure |
boolean |
false |
Listen on a non-loopback address without TLS. Frames travel unencrypted and unsigned: only for a network you fully trust, or behind a TLS proxy. |
hosts |
string[] |
the hosts joined at spawn() |
The host IDs of the deployment, in placement order. A host not listed is refused. Must be non-empty, unique and valid host IDs. |
minHosts |
number |
1 |
Without hosts: how many hosts must join before clusters are placed. Positive integer. |
joinTimeout |
number |
60000 |
Ms spawn() waits for the hosts before it rejects. |
lease |
number |
15000 |
Fencing lease in ms; at least 30. See Lease fencing. |
helloTimeout |
number |
3000 |
Ms a new connection has to authenticate before it is closed with 4408. |
maxUnauthenticated |
number |
16 |
Connections that may be waiting to authenticate at once. When full, the one that has waited longest is closed with 4429 to make room. |
maxUnauthenticatedPerAddress |
number |
4 |
Connections one client address may have waiting at once. Its oldest is closed with 4429 to make room. |
connectionsPerSecond |
number |
10 |
New connections accepted per second from one address (a token bucket with a burst of the same size). More are refused with HTTP 429 before the upgrade. |
clientAddress |
(request, peer) => string |
the peerâs IP | The address used by the two limits above. Behind a TLS proxy every host shares the proxyâs address: return the forwarded address the proxy sets (and only trust it from the proxy). |
local |
boolean |
false |
Also run clusters on the coordinatorâs machine (with script). |
moveClusters (0.3.2) |
boolean |
true |
Restart a lost hostâs clusters on another place once the lost host has certainly fenced itself. See Placement. false keeps them waiting for their host. |
| Condition | Error |
|---|---|
secret not a string of at least 16 characters |
RangeError |
port not an integer from 0 to 65535 |
RangeError |
tls without a non-empty cert and key |
TypeError |
non-loopback hostname without tls or allowInsecure |
RangeError |
lease below 30 or not finite |
RangeError |
an invalid ID in hosts, or hosts empty or with duplicates |
RangeError |
minHosts not a positive integer |
RangeError |
remote needs Bun on the coordinator: it listens with Bun.serve. A plain HTTP request to the port gets 426 with the body Lunibee cluster coordinator.
Coordinator members
Section titled âCoordinator membersâThese are ClusterManager members; the rest of the API is on the ClusterManager page.
| Member | Description |
|---|---|
remotePort |
The port hosts connect to, once the listener is open; undefined before. |
hosts() |
Every declared or connected host with its clusters, sorted by ID. See HostStatus. Empty without remote. |
health().clusters[].host |
Where each cluster runs: "local" or a host ID. |
hostJoin event |
(hostId: string) â a host authenticated, also on every reconnect. |
hostLeave event |
(hostId: string, reason: string) â a host disconnected, was lost or the coordinator stopped. Its clusters exit with signal SIGHUP and are handled as crashes (outside a shutdown). |
error event |
Also emitted when a host reports it could not start a worker (Host eu-1 could not run cluster 2: âŚ). |
Placement
Section titled âPlacementâWhen spawn() runs, the coordinator opens its listener and waits for the hosts:
- with
hosts, until every listed host has joined; - without it, until
minHostshosts have joined, then it uses the hosts connected at that moment, sorted by ID.
If they do not join within joinTimeout, spawn() rejects (Hosts did not join within 60000ms: [us-1]. or Only 1 of 2 hosts joined within 60000ms.) and the listener is closed again. A shutdown during the wait ends it without an error.
The placement list is those host IDs, with "local" in front when remote.local is true. Cluster n runs on placement[n % placement.length]. A cluster keeps its place for the life of the spawn: when it crashes it restarts on the same host. respawn() keeps the hosts connected and places again.
When a host is lost, its clusters exit and are restarted after the restart delay:
moveClusters: true(default, 0.3.2). A restart waits until the lost host has certainly fenced itself: two leases after the coordinator dropped it (the host kills its workers one lease after it last heard from the coordinator). Then the cluster starts on the connected place (a host, or"local"withremote.local) that runs the fewest clusters, andclusterMovedis emitted with{ from, to }. If the host reconnects before that, the cluster starts on it at once. A moved cluster stays where it is when its old host comes back. With no connected place, the restart fails and is retried like a crash.moveClusters: false. The restart is sent to the same host: while it is away the worker âexitsâ at once withSIGHUPand is retried after the restart delay, counting towardssupervisor.maxRestarts, until the host reconnects.
reassignOnGiveUp also moves shards across hosts: it hands a failed clusterâs shards to a ready cluster on any host.
ClusterHost
Section titled âClusterHostâThe agent that makes a machine part of a deployment.
new ClusterHost(options: ClusterHostOptions)Options
Section titled âOptionsâ| Option | Type | Default | Description |
|---|---|---|---|
url |
string | string[] |
â | The coordinatorâs address, wss://host:port. ws:// is accepted for this machine (localhost, 127.x.x.x, ::1), or for another one with allowInsecure. Must not contain credentials. A list (1â16, each checked the same way) names every coordinator of a deployment with a leader lock: the host tries them in order and moves to the next whenever one cannot be reached or drops it. |
allowInsecure |
boolean |
false |
Connect with ws:// to a coordinator on another machine, over a network you fully trust. |
secret |
string |
â | The coordinatorâs shared secret, at least 16 characters. |
id |
string |
â | This hostâs ID. Must be unique in the deployment. |
script |
string |
â | The worker script this host runs for every cluster. The coordinator cannot choose it. |
fork |
(script, env) => ChildProcess |
child_process.fork(script, [], { env }) |
Starts a worker. env is the hostâs environment plus the worker contract. |
reconnectDelay |
number |
1000 |
First delay before reconnecting, in ms. |
maxReconnectDelay |
number |
30000 |
Longest delay between reconnect attempts, in ms. |
connectAttempts |
number |
10 |
Attempts connect() makes while the coordinator closes the first connection as busy (4429) or slow (4408). |
The constructor throws a RangeError for an invalid id or a secret under 16 characters, and a TypeError for a missing script, a URL that is not absolute ws:// or wss://, a URL with a user name or password, or ws:// to another machine without allowInsecure. It does not connect.
Members
Section titled âMembersâ| Member | Description |
|---|---|
connect() |
Connects and resolves once the coordinator accepted this host. Rejects with The coordinator refused this host (<code>: <reason>). if the first attempt fails (bad secret, unknown or duplicate ID, version mismatch, or no coordinator at the address). A first attempt closed with 4429 or 4408 (the coordinator was busy) is retried with backoff, up to connectAttempts connections; the host is then closed and connect() can be called again. After a successful first connect, a lost connection is retried with backoff until close(). Called again while connecting or connected, it resolves at once. |
close() |
Kills every worker (SIGKILL), closes the connection with 1000 and stops reconnecting. Resolves when the socket has closed. A pending connect() rejects with The host was closed. |
connected |
Whether the coordinator has accepted this host and the link is open. |
clusters() |
Cluster IDs running on this host, sorted. |
on(event, listener), off(event, listener) |
Add or remove a listener. Return the host. A listener that throws is ignored. |
ClusterHostEvents:
| Event | Arguments | When |
|---|---|---|
connected |
â | The coordinator accepted this host (also after each reconnect). |
disconnected |
{ code: number; reason: string } |
An accepted connection ended. Every worker was killed first. |
clusterSpawn |
cluster: number |
A worker started for a cluster. |
clusterExit |
cluster: number, code: number | null, signal: string | null |
A worker exited. |
error |
error: Error |
A malformed frame from the coordinator (ignored), a worker process error, or a refusal after the first connect (The coordinator refused this host (4401: âŚ); the host keeps retrying). |
Reconnecting
Section titled âReconnectingâAfter an accepted connection is lost, the host kills its workers, emits disconnected and reconnects after reconnectDelay Ă (1 + 0â20 % jitter). Each failed attempt doubles the delay up to maxReconnectDelay; an accepted connection resets it. The host keeps retrying until close(), also when the coordinator refuses it; each refusal (4401, 4409, 4410, 4426) is reported as an error event.
What a host runs
Section titled âWhat a host runsâ- Only its own
script. The coordinator sends a cluster ID and an environment, never a path or code. - The environment is the hostâs own
process.envplus at most the seven worker-contract variables (SHARD_LIST,SHARD_COUNT,CLUSTER_ID,CLUSTER_COUNT,LUNIBEE_SHARD_BUS,LUNIBEE_CLUSTER_WORKER,LUNIBEE_IDENTIFY). A spawn request with any other variable (NODE_OPTIONS,LD_PRELOAD, a token) is rejected whole as a malformed frame. - The coordinator can send only
SIGTERM,SIGKILLorSIGINTto a worker. - One worker per cluster: a second spawn for a cluster that is still running is refused, and the coordinator reports it as an
errorand handles it as a crash. So is a worker that fails to fork, and a spawn request the host will not run (for example one with an environment variable outside the allowlist).
The workerâs IPC parent is the ClusterHost process. ClusterWorker exits a worker whose parent goes away (exitOnParentLoss), so killing the agent also stops its workers.
Authentication
Section titled âAuthenticationâEvery connection is authenticated before it can do anything:
- The coordinator sends
{ t: "challenge", nonce, version }with a fresh random 16-byte nonce. - The host checks
versionagainst its ownREMOTE_PROTOCOL_VERSION(4426on mismatch) and answers{ t: "hello", id, mac, version }, wheremacisHMAC-SHA256(secret, "<nonce>:<id>")in hex. The secret itself never travels. - The coordinator checks, in order: the version (
4426), the ID format and the MAC in constant time (4401for either, with the same reason, so a failure never says which), that the ID is inhostswhenhostsis set (4410), and that no host with that ID is connected (4409). - It answers
{ t: "welcome", leaseMs, pingMs }and emitshostJoin.
A connection that sends anything but a text hello first, or does not send it within helloTimeout, is closed (4400, 4408). At most maxUnauthenticated connections may wait at once, and maxUnauthenticatedPerAddress from one address; a new connection closes the longest-waiting one instead of being refused, so a flood of unauthenticated sockets cannot keep a real host out, and one address may open at most connectionsPerSecond per second. A flood from the same address as a real host (a shared NAT or proxy) can still delay it: put the coordinator behind a firewall or a TLS proxy that limits connections.
The HMAC proves who a host is. It does not encrypt or sign later frames: that is TLSâs job.
Lease fencing
Section titled âLease fencingâA shard must never run in two places, or Discord sees two sessions for it. The lease enforces that without a shared clock:
| Side | Rule |
|---|---|
| Coordinator | Pings every host every max(10, floor(lease / 3)) ms. Treats a host as lost only after two leases without any frame from it: closes it with 4504, emits hostLeave and lets its clusters restart. |
| Host | Kills every worker (SIGKILL) as soon as its connection closes, when the coordinator sends bye, and after one lease without any frame from the coordinator; then closes with 4504 and reconnects. |
The host has always fenced itself before the coordinator lets anything restart, so a cluster is never running on a host that has been written off. The lease comes from the coordinator (welcome.leaseMs), so both sides use the same value.
Every worker start has an epoch number. Frames and exits from an earlier process of the same cluster are ignored on both sides, so a late message from a replaced worker never reaches its successor.
Coordinator failover
Section titled âCoordinator failoverâ(0.3.2) Run two or more coordinators with the same options and a shared leader lock; only the holder runs clusters, and the hosts follow whichever one leads.
import { ClusterManager, RedisLeaderLock } from "lunibee";
const manager = new ClusterManager({ token: process.env.DISCORD_TOKEN!, script: "./worker.ts", remote: { port: 7000, secret: process.env.CLUSTER_SECRET!, hosts: ["us-1", "us-2"] }, leader: { lock: new RedisLeaderLock({ client: new Bun.RedisClient(process.env.REDIS_URL!) }), ttl: 10_000 },});await manager.spawn(); // resolves once this coordinator leads and started the clustersOn each host: new ClusterHost({ url: ["wss://coord-a:7000", "wss://coord-b:7000"], ⌠}).
| Step | What happens |
|---|---|
| Standby | spawn() tries the lock every ttl / 2 ms. It does not listen for hosts, so they cannot join it. |
| Leading | The holder renews the lock every ttl / 4 ms. When it takes the lock it emits leaderAcquired and waits two remote.leases (so hosts of a previous leader have fenced), then opens the listener and starts clusters as usual. |
| Losing it | If the lock cannot be renewed for ttl / 2 ms, or another coordinator holds it, the leader kills every cluster at once (SIGKILL, local and remote), closes the listener, emits leaderLost and becomes a standby again. That happens by 3/4 Ă ttl after its last renewal, before the lock can lapse. |
| Hosts | A host whose coordinator went away kills its workers and tries the next URL of its list until a leader accepts it. |
So a shard is never owned by two coordinators: the old leader stops before the lock lapses, and the new one waits for the old leaderâs hosts to fence. shutdownAll() and killAll() give the lock back, so a standby leads at once; respawn() keeps it.
RedisLeaderLock({ client, key?, token? }) keeps the lock in one Redis key (default lunibee:cluster:leader) with an expiry; taking, extending and releasing are single Lua scripts, with the key and token passed as KEYS/ARGV. client is anything with send(command, args): Bun.RedisClient as it is, or { send: (c, a) => redis.call(c, ...a) } for ioredis. Any object with acquire(ttlMs): Promise<boolean> and release(): Promise<void> works as a LeaderLock; acquire must take the lock when it is free and extend it when this process holds it, atomically.
Hosts and HostStatus
Section titled âHosts and HostStatusâclusters.hosts(); // HostStatus[]| Field | Type | Description |
|---|---|---|
id |
string |
Host ID. |
connected |
boolean |
Whether its socket is open and authenticated. |
lastSeen |
number |
When it last sent anything (epoch ms); 0 when not connected. |
clusters |
number[] |
Clusters running on it now, sorted. |
The list holds every host in remote.hosts and every connected host, sorted by ID.
Host IDs
Section titled âHost IDsâ1 to 64 characters: letters, digits, ., _ and -.
RemoteClose
Section titled âRemoteCloseâWebSocket close codes between a coordinator and a host:
| Code | Name | Meaning |
|---|---|---|
4400 |
BadRequest |
A binary frame, a malformed frame, or something other than hello first. Also for an authenticated host that sends a malformed frame. |
4401 |
AuthFailed |
Invalid host ID or wrong MAC. |
4408 |
HelloTimeout |
No hello within helloTimeout. |
4409 |
DuplicateHost |
A host with this ID is already connected. |
4410 |
UnknownHost |
The ID is not in remote.hosts. |
4426 |
VersionMismatch |
The two sides speak different protocol versions. |
4429 |
TooManyConnections |
Closed to make room: the connection had waited longest while maxUnauthenticated (or its addressâs maxUnauthenticatedPerAddress) connections were waiting. |
4500 |
Shutdown |
The coordinator stopped (shutdownAll(), killAll()). |
4504 |
HostLost |
Sent by the coordinator after two silent leases, or by the host after one. |
REMOTE_PROTOCOL_VERSION is 1. A coordinator and a host must have the same version; upgrade both together.
Security defaults
Section titled âSecurity defaultsâ- Loopback by default. The coordinator binds
127.0.0.1unlesshostnamesays otherwise. - Encryption off this machine. Listening on any other address needs
tls, or an explicitallowInsecure: true. Behind a TLS-terminating proxy, bind127.0.0.1and let the proxy forward. AClusterHostlikewise refusesws://to another machine withoutallowInsecure. - No secret on the wire. Hosts answer an HMAC challenge; the secret is at least 16 characters on both sides.
- Bounded input. Frames are text only and at most 8 MiB in UTF-8 bytes on both sides (
maxPayloadLengthon the listener, and checked again when parsed).maxMessageByteson theClusterManager(default 8 MiB) bounds what the coordinator relays from any worker, local or remote. Unauthenticated connections are capped and timed out. - Host-owned execution. The host chooses the script; the coordinator can set only the worker-contract variables and send three signals.
- No token transfer. Workers read the bot token from their hostâs environment.
- Without a leader lock the coordinator is a single point of failure: if it stops, hosts fence their workers. Run it under a process supervisor, or run standbys.
- Failover relies on timers: a leader whose event loop is blocked for longer than half the lock
ttlcannot stand down in time. Keepttlwell above your longest pause (garbage collection, a busy loop). - A moved cluster does not move back when its host returns.
- The coordinator needs Bun (
Bun.serve). AClusterHostneeds a globalWebSocketandchild_process.fork(Bun, or Node.js 22 or later).