Skip to content

ClusterManager

ClusterManager (from @lunibee/sharding) runs a bot as several processes. It:

  • splits the shards across clusters and forks one worker process per cluster;
  • hands out IDENTIFY slots for the whole deployment, so Discord’s limit holds across processes;
  • routes ShardBus messages to the cluster that owns the target shard;
  • answers requests, broadcasts and health queries for the whole deployment;
  • restarts crashed clusters with backoff, and can move a failed cluster’s shards to a healthy one;
  • with remote, places clusters on other machines (see ClusterHost).

ClusterManager orchestrates processes; ShardManager runs shards. The manager owns the shard-to-cluster map and keeps the workers alive; a ShardManager inside each worker opens the Gateways for that cluster’s shards. A shard is never owned by two clusters at once. The sharding guide explains how the two fit together.

Works under Bun and Node.js (child_process.fork() with IPC). The remote option needs Bun on the coordinator.

import { ClusterManager, ClusterWorker, readClusterContext } from "lunibee";
// or the package on its own:
import { ClusterManager, ClusterWorker, readClusterContext } from "@lunibee/sharding";

Two scripts: the parent, which runs the ClusterManager, and the worker, which the parent forks once per cluster.

// index.ts (parent)
import { ClusterManager } from "lunibee";
const clusters = new ClusterManager({
token: process.env.DISCORD_TOKEN!,
script: "./worker.ts",
shardCount: "auto",
shardsPerCluster: 4,
waitForReady: true,
startTimeout: 120_000,
supervisor: { restartDelay: 1000, backoff: 2, jitter: 0.2, maxRestarts: 5 },
reassignOnGiveUp: true,
});
clusters.on("clusterReady", (cluster) => console.log(`cluster ${cluster.id} ready`));
clusters.on("clusterRestart", (cluster, { delay, attempt }) =>
console.warn(`cluster ${cluster.id} restarts in ${delay}ms (#${attempt})`),
);
clusters.on("error", (error) => console.error(error));
await clusters.spawn();
process.on("SIGTERM", () => void clusters.shutdownAll());
// worker.ts (one process per cluster)
import { ShardManager, ShardBus, GatewayIntentBits } from "lunibee";
// Inside a cluster, ShardManager reads the worker contract from the
// environment: it runs only this cluster's shards, asks the parent for every
// IDENTIFY, and reports ready and health to the parent.
const shards = new ShardManager({
token: process.env.DISCORD_TOKEN!,
intents: [GatewayIntentBits.Guilds],
});
for (const id of shards.shardIds)
new ShardBus(id, "my-bot").respond("guildCount", () => 42);
await shards.connect();

The worker inherits the parent’s environment, so it reads the same DISCORD_TOKEN.

new ClusterManager(options: ClusterManagerOptions)

The constructor validates the options and throws; it does not fork or call Discord. spawn() does that.

Option Type Default Description
token string — Bot token. Required; an empty one throws a TypeError.
script string — Worker script, forked once per cluster on this machine. Required (also with remote, where it is used only for clusters on this machine); an empty one throws a TypeError.
shardCount number | "auto" unset Total shards. "auto" asks /gateway/bot. When unset, spawn() also asks Discord, but autoScaleInterval only acts with an explicit "auto".
guildsPerShard number unset A sizing hint for "auto": at least ceil(guilds / guildsPerShard) shards, and never fewer than Discord recommends. Integer from 1 to 2500 (Discord’s limit); Discord recommends about 1000. It only picks the shard count: Discord puts each guild on shard (guild_id >> 22) % shardCount, so a shard gets about that many guilds, not exactly. Throws with a numeric shardCount.
guildCount number asked from Discord The guild count for guildsPerShard, instead of reading approximate_guild_count from /applications/@me.
shardIds number[] every shard The shards this deployment runs, out of shardCount. Must be unique integers in [0, shardCount); checked in the constructor when shardCount is a number, otherwise in spawn().
clusterCount number logical CPUs (os.cpus().length) Clusters to split the shards across, in contiguous blocks: 10 shards over 3 clusters are [0-3] [4-6] [7-9]. Positive integer. More clusters than shards gives one shard per cluster (the plan event reports the count asked for). On a one-CPU machine the default is one cluster: set clusterCount or shardsPerCluster to be sure.
shardsPerCluster number unset Shards per cluster; the cluster count is ceil(shards / shardsPerCluster) and the last cluster may have fewer: 10 shards at 4 per cluster are [0-3] [4-7] [8-9]. Positive integer. Setting it together with clusterCount throws a RangeError.
Option Type Default Description
maxConcurrency number Discord’s value when fetched, else 1 Discord’s max_concurrency: how many IDENTIFY rate-limit keys there are. Positive integer. When /gateway/bot was read, this can only lower Discord’s value.
fetchLimits boolean false Read max_concurrency and the session-start budget from /gateway/bot even with a numeric shardCount. A failed request is reported as an error event and the start continues without them.
startup (0.4.0) "safe" | "sequential" "safe" "safe": shards start as fast as the IDENTIFY rules allow, different rate-limit keys in parallel. "sequential": one shard at a time across all clusters; each holds its slot until it is READY (or identify.holdTimeout, which workers also use as their READY wait), then the next waits identify.interval. Slow, for debugging and small bots. Needs identify.coordinate.
identify.coordinate boolean true Hand out IDENTIFY slots from the parent for every cluster. false lets each cluster pace its own IDENTIFYs (LUNIBEE_IDENTIFY=local).
identify.interval number 5000 Milliseconds between two IDENTIFYs on one key.
identify.holdTimeout number 60000 A granted slot that a worker never gives back is released after this many ms.
spawnInterval number 500 Milliseconds between forking two clusters, to spread the CPU cost of starting. IDENTIFY pacing does not depend on it. Cannot be negative.
Option Type Default Description
waitForReady boolean false Make spawn() resolve only once every cluster reported ready.
startTimeout number off A cluster that has not reported ready this many ms after its fork is killed (SIGKILL) and restarted like a crash. Off by default, because a worker that never reports ready would be killed. Cannot be negative. Also the timeout of the take-over request in reassignment (120000 when unset).
requestTimeout number 5000 Default timeout of request(), broadcast() and requestCluster(), in ms. Cannot be negative.
maxPendingRequests number 10000 Most requests that may wait for answers at once. Positive integer. Beyond it, a new request throws.
maxMessageBytes number 8388608 (8 MiB) Largest payload the manager handles, in UTF-8 bytes. See Payload limits. Positive integer.
Option Type Default Description
restartOnExit boolean true Re-fork a cluster that exits unexpectedly, with the same shards. false leaves it failed.
restartDelay number 5000 Delay before the first restart, in ms. supervisor.restartDelay wins when both are set.
supervisor SupervisorOptions fixed 5 s delay, no limit Backoff, jitter and a give-up limit. See Restarts.
reassignOnGiveUp boolean false When a cluster gives up, move its shards to the ready cluster that owns the fewest. See Reassignment.
shutdownTimeout number 5000 Grace period after SIGTERM before SIGKILL, in ms, for shutdownAll() and respawn().
onClusterExit (cluster, code, signal) => void none Called whenever a cluster process exits, before any restart. The clusterExit event carries the same and more.
onGiveUp (cluster) => void none Called when a cluster will not be restarted. Same as the clusterGiveUp event.

A callback or listener that throws is ignored; it cannot break supervision.

Option Type Default Description
autoScaleInterval number off With shardCount: "auto", check the recommended count every this many ms and respawn() when it changed. Integer of at least 1000.
onAutoScaleError (error: Error) => void none Called when a background auto-scale check fails. The next tick retries.
fork (script, env) => ChildProcess child_process.fork(script, [], { env }) Starts a local worker. env is the parent’s environment plus the worker contract. Replace it for tests or a custom launcher; the returned process needs IPC (send(), message, exit). threadFork() runs clusters as worker threads.
leader (0.3.2) { lock: LeaderLock; ttl?: number } off Several coordinators, one active: spawn() waits until this one holds the lock. ttl defaults to 10000 (at least 100). See Coordinator failover.
remote RemoteOptions off Accept ClusterHost agents on other machines and place clusters on them.
Condition Error
token or script missing or blank TypeError
clusterCount and shardsPerCluster both set RangeError
clusterCount, shardsPerCluster or maxConcurrency not a positive integer RangeError
maxMessageBytes or maxPendingRequests not a positive integer RangeError
spawnInterval, startTimeout or requestTimeout negative RangeError
autoScaleInterval not an integer of at least 1000 RangeError
guildsPerShard outside 1–2500, or set with a numeric shardCount RangeError
shardIds duplicate or outside [0, shardCount) RangeError
supervisor values out of range RangeError (see Restarts)
invalid remote options RangeError / TypeError (see ClusterHost)
Property Type Description
clusters Map<number, ClusterInfo> Live clusters (starting or ready) by ID. A crashed cluster leaves the map until it is restarted.
shardCount number Total shards of the deployment. 0 before the first spawn().
remotePort number | undefined The port hosts connect to, once the listener is open. Useful with remote.port: 0.
isLeader (0.3.2) boolean Whether this coordinator holds the leader lock. Always true without leader.
spawn(): Promise<void>

Resolves the shard count, allocates shards, waits for hosts (with remote) and forks every cluster, spawnInterval apart. With waitForReady it then waits for readiness.

  • Calling it again while clusters run, or while the first call is still starting, returns the same promise.
  • Called during a shutdown, it waits for the shutdown and then starts.
  • A failed start (Discord unreachable, hosts that did not join) rejects and can be retried.
  • Inside a cluster worker it rejects: a worker must never fork the fleet again.
ready(timeoutMs?: number): Promise<void>

Resolves when every cluster is ready. Rejects after timeoutMs if given, or when the manager starts stopping. There is no timeout by default.

shutdownAll(timeoutMs = shutdownTimeout): Promise<void>

Graceful stop. Cancels restart timers, start timers, pending requests, IDENTIFY waits and ready() waiters; sends SIGTERM to every cluster, and SIGKILL to those still running after timeoutMs; closes the host listener. Safe to call twice or concurrently. Exits during a shutdown are never restarted. Every cluster ends stopped.

killAll(): void

Immediate stop. Cancels the same work as shutdownAll(), calls kill() (SIGTERM) on every cluster process without waiting, closes the host listener (hosts then kill their own workers), and marks every cluster stopped. Prefer shutdownAll().

respawn(newShardCount: number): Promise<void>

Shuts the clusters down gracefully, forgets their crash history and spawns again with the new total. Hosts stay connected.

checkAutoScale(): Promise<void>
fetchRecommendedShardCount(): Promise<number>

checkAutoScale() runs one auto-scale check: with shardCount: "auto" it compares Discord’s count (or the guildsPerShard count) with shardCount and calls respawn() when they differ. It does nothing otherwise, and passes failures to onAutoScaleError instead of throwing. fetchRecommendedShardCount() returns that count.

clusterForShard(shardId: number): ClusterInfo | undefined
assignments(): ReadonlyMap<number, number>

The cluster that owns a shard (whatever its state), or undefined when the shard is not part of the deployment; and a copy of the whole shard → cluster map. Ownership changes only through reassignment.

request<R = unknown, T = unknown>(shardId: number, type: string, data: T, options?: RequestOptions): Promise<R>

Sends a request to one shard, routed to the cluster that owns it, and resolves with the result of that shard’s ShardBus.respond() handler.

Rejects with When
RangeError data is larger than maxMessageBytes, or the shard is not part of the deployment.
Error("Shard 3 is down: cluster 1 is restarting.") The owning cluster is not running. Fails at once.
Error("Cluster 1 exited.") The owning cluster exited while the request waited.
Error(<handler message>) The handler threw.
Error('Shard 3 did not reply to "stats" in time.') No answer within the timeout, or signal aborted.
broadcast<R = unknown, T = unknown>(type: string, data: T, options?: RequestOptions): Promise<BroadcastResult<R>>

Asks every shard of every live cluster. It never rejects for a slow, failed or down shard (only a RangeError for oversized data, or when maxPendingRequests is reached). It settles as soon as every expected shard answered, or at the timeout.

const { replies, missing } = await clusters.broadcast<number>("guildCount", null, { timeoutMs: 5000 });
const total = replies.reduce((sum, r) => sum + (r.result ?? 0), 0);
requestCluster<R = unknown, T = unknown>(clusterId: number, type: string, data: T, options?: RequestOptions): Promise<R>

Sends a request to a worker process (not a shard), answered by a ClusterWorker.onRequest() handler. Rejects with a RangeError for an unknown cluster or oversized data; with Error("Cluster 1 is restarting.") when it is not running; with the handler’s error message; or with Error('Cluster 1 did not reply to "flush" in time.').

await clusters.requestCluster(1, "flushCaches", null);
health(): ClusterManagerHealth
refreshHealth(options?: { metrics?: boolean; timeoutMs?: number }): Promise<ClusterManagerHealth>

health() returns what the workers last reported; nothing is sent. refreshHealth() asks every live cluster for a fresh report first (timeout 2000 ms per cluster; process metrics only with metrics: true). A cluster that does not answer keeps its last report and is not treated as failed. See Health.

hosts(): HostStatus[]

The hosts of the deployment, declared or connected, with the clusters on each. Empty without remote. See ClusterHost.

Add or remove a listener for one of the events. Both return the manager.

Field Type Description
timeoutMs number Milliseconds to wait. Defaults to requestTimeout (5000).
signal AbortSignal Stops waiting. An aborted request settles like a timeout.
Field Type Description
replies ShardReply<R>[] One entry per shard that answered or failed: { shardId, result } or { shardId, error }. A shard whose cluster exited (or could not be sent to) during the broadcast is here with an error.
missing number[] Shards that did not answer, sorted: their cluster was not running when the broadcast started, or they did not answer before the timeout or abort.

The object is live: the manager updates it as the cluster changes.

Field Type Description
id number Cluster ID.
process ChildProcess The current process (the last one while a restart is pending). For a cluster on another host it is a proxy whose pid is the pid on that host.
host string "local" or the ID of the host it runs on.
shards number[] Shards it owns.
state ClusterState See Cluster states.
restarts number Restarts after a crash since spawn().
startedAt number When the current process was forked (epoch ms).
readyAt number? When the current process reported ready.
lastReadyAt number? When any process of this cluster last reported ready.
report WorkerReport? The worker’s last health report.

ClusterState is one of:

State Meaning
starting Forked; has not reported ready.
ready The worker reported its shards ready.
restarting Exited unexpectedly; a restart is scheduled.
stopping A deliberate shutdown is in progress.
stopped Stopped on purpose (shutdownAll(), killAll(), respawn()). Final.
failed Crashed too often (or restartOnExit: false) and will not be restarted.
starting ──ready──▶ ready
│ │
└──── crash ──────┴──▶ restarting ──delay──▶ starting
│
limit reached ──▶ failed
any live state ── shutdownAll()/killAll() ──▶ stopping ──▶ stopped

A startTimeout that passes while starting kills the process; its exit is handled as a crash.

Event Arguments When
clusterSpawn cluster: ClusterInfo A cluster process was forked (also for each restart).
clusterReady cluster: ClusterInfo A cluster’s worker reported ready.
ready — Every cluster is ready. Emitted again after a recovery brings them all back.
clusterExit cluster, code: number | null, signal: NodeJS.Signals | null, expected: boolean A cluster process exited. expected is true for a deliberate shutdown.
clusterRestart cluster, { delay: number; attempt: number } A restart was scheduled delay ms from now; attempt is the cluster’s restarts count.
clusterUnstable cluster, { attempt: number; delay: number } Emitted with every restart once the cluster has been restarted three or more times since spawn().
clusterStartTimeout cluster: ClusterInfo The cluster did not report ready within startTimeout and is being killed.
clusterGiveUp cluster: ClusterInfo The cluster exceeded supervisor.maxRestarts and will not be restarted.
shardsReassigned { shardIds: number[]; from: number; to: number } A failed cluster’s shards were moved to another cluster.
hostJoin hostId: string A host authenticated (also when it reconnects).
hostLeave hostId: string, reason: string A host disconnected or was lost. Its clusters exit and are handled as crashes.
clusterMoved (0.3.2) cluster, { from: string; to: string } A lost host’s cluster is restarting on another place (remote.moveClusters).
leaderAcquired (0.3.2) — This coordinator took the leader lock.
leaderLost (0.3.2) reason: string This coordinator lost the lock, killed its clusters and is a standby again.
plan (0.4.0) ClusterPlanInfo Each start: totalShards, clusterCount, clusterCountSource ("clusterCount", "shardsPerCluster" or "default"), requestedClusterCount, clusters ({ id, shards }), maxConcurrency, startup, identifyCoordinated. Before any process is forked.
identify (0.4.0) IdentifyEvent The IDENTIFY queue: { type, owner, shardId, bucket, waitedMs? }, type one of "queued", "granted", "released", "expired" (a slot not given back in holdTimeout), "dropped" (its cluster exited). bucket is shardId % maxConcurrency.
shutdown (0.4.0) { clusters: number; durationMs: number } shutdownAll() finished: every worker exited, or was killed after the timeout.
identifyWait { shardId: number; waitMs: number } An IDENTIFY waits for the session-start budget to reset. Repeated waits for the same shard and second are not re-emitted.
error error: Error Something went wrong that did not stop the deployment. See below.

error is emitted for: a cluster whose process could not be forked (the message names the cluster and its shards; it goes through the restart policy like a crash, and the clusters after it still start); a malformed control frame from a worker; a ready report naming shards the cluster does not own; an oversized shard-bus message or health report; an asynchronous IPC send failure of a worker process; a failed fetchLimits request; a reassignment that found no target or was refused; and, with remote, a host that could not start a worker. With no error listener these are dropped.

A cluster is starting until its worker sends a ready frame, which a ShardManager does once every one of its shards finished its handshake (and again after a shard dropped and recovered). Workers without a ShardManager call ClusterWorker.ready() themselves. A ready frame that names a shard the cluster does not own is rejected with an error event.

When every cluster in the deployment is ready, ready() waiters resolve and the ready event is emitted. A crash clears that state, so ready is emitted again after recovery. Readiness means every shard is served by a ready cluster: a failed cluster whose shards were reassigned no longer counts. When a cluster fails and its shards are left with no owner (no reassignOnGiveUp, restartOnExit: false, no ready cluster to take them, or the takeover failed), pending ready() calls reject with an error naming the cluster and its shards instead of waiting forever.

Discord allows one IDENTIFY per rate-limit key (shard_id % max_concurrency) every five seconds, for the whole bot, and a daily session-start budget. With identify.coordinate on (the default), the parent holds both rules for every cluster, local or remote:

  1. Before a worker connects a shard, it asks the parent for a slot (ClusterWorker.identifyGate.acquire(shardId)). ShardManager does this for you, and so does a Client created with gateway: { shardId, shardCount } inside a worker (0.4.0); its re-IDENTIFYs after a lost session ask too.
  2. The parent grants one slot per key at a time, at least identify.interval apart, in request order.
  3. Once the IDENTIFY went out, the worker gives the slot back (release(shardId)) and the key rests for one interval. A slot not given back within identify.holdTimeout is released for the worker.

The parent denies a request for a shard the cluster does not own, and a second request for a shard that is already waiting, so a worker cannot queue more requests than it has shards. When a cluster exits, its waiting requests are dropped and the slots it held are freed.

The budget is known when /gateway/bot was read (shardCount: "auto", unset, or fetchLimits: true). Each grant spends one session; when none are left, requests wait for Discord’s reset_after (an identifyWait event says so) instead of failing, and the budget refills every 24 hours after that. Without a known budget the parent paces but never blocks.

With identify.coordinate: false, workers get LUNIBEE_IDENTIFY=local and pace themselves; a request for a slot is denied with "IDENTIFY coordination is off.". A RESUME, or a reconnect handled inside one Gateway, does not use a slot.

IdentifyCoordinator is the class the parent uses. It is exported for custom orchestrators:

import { IdentifyCoordinator, IDENTIFY_INTERVAL } from "lunibee";
const slots = new IdentifyCoordinator({ maxConcurrency: 16, interval: IDENTIFY_INTERVAL });
await slots.acquire(/* owner */ 0, /* shardId */ 5);
// ...send IDENTIFY...
slots.release(0, 5);
Option Default Description
maxConcurrency 1 Number of keys. Positive integer.
interval 5000 Ms between two IDENTIFYs on one key. Cannot be negative.
budget none { total, remaining, resetAfter, period? }; period defaults to 24 h.
holdTimeout 60000 Ms after which an unreleased grant is released. Must be positive.
onBudgetWait none ({ shardId, waitMs }) => void when a request waits for the budget.
now, schedule, cancel Date.now, setTimeout, clearTimeout Clock and timer, for tests.

Members: acquire(owner, shardId), release(owner, shardId) (ignored unless owner holds that slot), drop(owner, reason?) (rejects the owner’s waiters and frees its slots), close(), pump(), and the read-only maxConcurrency, remaining, waiting and holding.

Workers send ShardBus messages to the parent over IPC (the worker contract sets LUNIBEE_SHARD_BUS=ipc). The parent relays only between clusters; buses in the same worker deliver to each other directly (IpcTransport delivers locally as well as sending to the parent).

Target What the parent does
A shard ID Sends it only to the cluster that owns the shard. If the target is in the sender’s own cluster, or not part of the deployment, nothing is relayed. If the owner is not running, the sender gets a peer-down notice and its pending request to that shard fails at once.
null (broadcast) Sends it to every other live cluster.
the parent (-1) Consumed as an answer to request() or broadcast(). Answers count only from the cluster that owns the answering shard, once per shard.

A bus frame from a worker is dropped silently unless:

  • type is a string of 1 to 256 characters;
  • id is a string of at most 256 characters;
  • source is an integer;
  • target is an integer or null.

Only the documented envelope is relayed (type, id, source, target, data and expectsReply when present); other fields are stripped. Anything on the IPC channel that is not a bus or cluster control frame is the bot’s own process.send() traffic and is ignored. A control frame that does not parse is reported as an error event.

maxMessageBytes (default 8 MiB, the same as the host link) is measured in UTF-8 bytes: a string as it is, anything else as its JSON text. A value that cannot be serialized (a cycle) counts as oversized.

Payload Over the limit
data of a shard-bus message from a worker Dropped, not relayed; an error event says so. A request waiting for it times out as usual.
A worker’s answer to requestCluster() The request settles at once with an error.
A health report from a worker Ignored; an error event says so. The previous report is kept.
data passed to request(), broadcast() or requestCluster() The call rejects with a RangeError; nothing is sent.

The parent’s table of pending requests, exported for custom orchestrators. Each request waits for a set of keys, each owned by one cluster, and settles when all answered, at its timeout, or when its signal aborts. An answer is accepted only from the key’s owner and only once; late, duplicate and unknown answers are ignored. A timer per request guarantees the entry is removed.

import { RequestTracker } from "lunibee";
const tracker = new RequestTracker("parent", 10_000); // id prefix, maxPending
const { id, done } = tracker.begin({ keys: new Map([["c:1", 1]]), timeoutMs: 5000 });
tracker.answer(id, "c:1", 1, { result: "ok" }); // true when accepted
const { replies, missing } = await done; // [{ key, result?, error? }], ["..."]

failOwner(owner, error) settles every unanswered key of a cluster with error; close() settles everything with what it has; size is the number of waiting requests. begin() throws when maxPending requests are already waiting.

Field Type Description
state "idle" | "starting" | "ready" | "degraded" | "stopping" | "stopped" idle before the first spawn(); stopping during a shutdown; stopped when every cluster is stopped; ready when all are ready; degraded when any is restarting or failed; otherwise starting.
totalShards number shardCount.
clusters ClusterHealth[] Every cluster of the deployment, sorted by ID, whatever its state.
readyClusters number Clusters in ready.
missingShards number[] Shards whose owning cluster is not ready, sorted.
Field Type Description
id number Cluster ID.
state ClusterState Lifecycle state.
pid number? Process ID (on its host, for a remote cluster); only while starting or ready.
host string "local" or the host ID.
shardIds number[] Shards it owns now.
uptimeMs number Ms since the current process was forked; 0 when it is not running.
restarts number Restarts after a crash.
startedAt number? When the current process was forked.
readyAt number? When the current process reported ready.
lastReadyAt number? When any process of this cluster last reported ready.
shards ShardHealth[] Per-shard health from the last report; empty until the worker reports.
metrics WorkerMetrics? { rssBytes, heapUsedBytes, cpuUserMicros, cpuSystemMicros }, when requested with refreshHealth({ metrics: true }) and the worker answered.
reportedAt number? When shards and metrics were reported (epoch ms).

ShardHealth is { id, state, ping, status, readyAt?, disconnectedAt? }, where state is the Gateway state and status the shard lifecycle (idle, connecting, ready, disconnected, unstable, destroyed). See ShardManager.

A worker’s ShardManager pushes a report each time all its shards become ready, and answers refreshHealth(). Cluster health follows the process: a Gateway problem in one shard, a reconnect storm included, never restarts a healthy cluster. Only an exited process (or a missed startTimeout) is restarted.

When a cluster exits outside a shutdown, the parent:

  1. drops its IDENTIFY requests and frees its slots;
  2. settles its pending requests with "Cluster <id> exited.";
  3. calls onClusterExit and emits clusterExit;
  4. tells the other clusters its shards are down, so their pending requests to those shards fail at once;
  5. schedules a restart with the same shards (clusterRestart), or gives up.

The supervisor option (SupervisorOptions) decides the delay:

Option Default Description
restartDelay restartDelay, else 5000 Delay before the first restart, in ms. Cannot be negative.
backoff 1 Each further crash inside window multiplies the delay by this. At least 1; 1 is a fixed delay.
maxDelay 300000 Longest delay, in ms. Cannot be negative.
jitter 0 Random spread added to each delay, as a fraction of it (0–1). The result is still capped at maxDelay.
maxRestarts unlimited Give up after this many restarts inside window. 0 gives up on the first crash.
window 60000 The time, in ms, over which restarts are counted. A cluster that stays up longer is forgiven. Must be positive.

The delay of the n-th restart inside the window is min(maxDelay, restartDelay × backoff^(n−1)) × (1 + jitter × random), capped at maxDelay. When the limit is reached the cluster becomes failed, onGiveUp is called and clusterGiveUp is emitted. With restartOnExit: false an exit makes the cluster failed straight away, without clusterGiveUp.

A restarted process is a new one: late messages and the exit of the process it replaced are ignored.

ShardSupervisor is the class behind this, exported for custom orchestrators: new ShardSupervisor(options?, random?), next(id, now?) returns the delay in ms or null to give up, and reset(id?) forgets crashes.

With reassignOnGiveUp: true, a cluster that gives up hands its shards to the ready cluster that owns the fewest:

  1. The parent asks that cluster’s worker to start them (a built-in requestCluster() request that ShardManager answers with addShards()), with a timeout of startTimeout, or 120000 ms.
  2. Only after the worker confirms does ownership move: clusterForShard() returns the new owner, the target’s shards grows, the failed cluster’s shards becomes empty and shardsReassigned is emitted.

The failed process is already dead and ownership moves only after confirmation, so no shard ever has two owners. If no cluster is ready, or the worker refuses or times out, an error event says so and the shards stay down. Reassignment does not happen for restartOnExit: false.

(0.3.2) fork: threadFork(options?) runs every local cluster as a worker thread of the manager’s process instead of a forked process. The worker script does not change: ShardManager, ShardBus and ClusterWorker find the parent through the thread’s port, and IDENTIFY coordination, routing, health and restarts work as with processes.

import { ClusterManager, threadFork } from "lunibee";
const manager = new ClusterManager({
token: process.env.DISCORD_TOKEN!,
script: "./worker.ts",
shardsPerCluster: 1, // one shard per thread
fork: threadFork({ resourceLimits: { maxOldGenerationSizeMb: 512 } }),
});
  • Threads start faster and share one process’s memory overhead. A crash that ends the process (out of memory, a native fault) ends every cluster with it, and ClusterInfo.process.pid is the thread ID.
  • kill() terminates the thread: a thread cannot handle signals, so shutdownAll() ends it at once, as SIGTERM ends a process without a handler. Its exit reports the signal it was asked for.
  • env becomes the thread’s process.env. Remote hosts still fork processes.
  • threadEndpoint() returns the parent link inside such a thread (undefined elsewhere), for code that talks to the parent itself.

Process per shard: shardsPerCluster: 1 gives every shard its own process (or thread).

ClusterManager sets these in every worker, on top of the parent’s environment (local clusters) or the host’s environment (remote clusters):

Variable Value
LUNIBEE_CLUSTER_WORKER 1. Marks a worker. A ClusterManager refuses to spawn() in such a process.
CLUSTER_ID This cluster’s ID.
CLUSTER_COUNT How many clusters the deployment started (clusters with shards).
SHARD_COUNT Total shards of the deployment (Discord’s num_shards).
SHARD_LIST Comma-separated shard IDs this cluster owns.
LUNIBEE_SHARD_BUS ipc: ShardBus uses the IPC channel instead of a BroadcastChannel.
LUNIBEE_IDENTIFY coordinated when the parent hands out IDENTIFY slots, otherwise local.
LUNIBEE_STARTUP (0.4.0) sequential with startup: "sequential"; unset otherwise.
LUNIBEE_READY_TIMEOUT (0.4.0) With startup: "sequential": identify.holdTimeout (default 60000), how long a worker waits for a shard’s READY.

These seven are the only variables a coordinator can set in a worker on another host.

readClusterContext(env = process.env): ClusterContext | null
isClusterWorker(env = process.env): boolean

isClusterWorker() is true when LUNIBEE_CLUSTER_WORKER is 1. readClusterContext() parses the variables above into a ClusterContext, returns null outside a worker, and throws a RangeError when they are present but malformed (not non-negative integers, or shard IDs outside SHARD_COUNT or repeated).

ClusterContext field Type From
clusterId number CLUSTER_ID
clusterCount number CLUSTER_COUNT
shardCount number SHARD_COUNT
shardIds number[] SHARD_LIST
coordinatedIdentify boolean LUNIBEE_IDENTIFY === "coordinated"
startup "safe" | "sequential" LUNIBEE_STARTUP (default "safe")
readyTimeout number | undefined LUNIBEE_READY_TIMEOUT

The worker side of the contract: it reports readiness and health to the parent, answers the parent’s requests, and asks the parent for IDENTIFY slots. A ShardManager inside a cluster uses it automatically. Use it directly when a worker runs its own Gateways, or to answer requestCluster().

import { ClusterWorker } from "lunibee";
const worker = ClusterWorker.current(); // undefined outside a cluster
worker?.onRequest("flushCaches", async () => {
await cache.flush();
return { ok: true };
});
static current(): ClusterWorker | undefined
new ClusterWorker(context: ClusterContext, endpoint?: WorkerEndpoint, options?: { exitOnParentLoss?: boolean; exit?: (code: number) => void })

current() returns this process’s worker, creating it on first use, or undefined when the process was not forked by a ClusterManager (or has no IPC channel). It throws when the contract variables are malformed. ClusterWorker.reset() closes and forgets it (tests).

The constructor takes a context, an endpoint (process by default: anything with send, on and off) and options. It throws if the endpoint has no send().

Option Default Description
exitOnParentLoss true for the real process, false for an injected endpoint Exit with code 1 when the parent’s IPC channel disconnects. Without a parent nothing coordinates IDENTIFY or fences the process, and its Gateway sessions would keep running beside the replacement the parent starts. A closed worker never exits this way.
exit process.exit Replaces the exit call; for tests.

Losing the parent always rejects pending IDENTIFY requests, whatever this option says.

Member Description
context The ClusterContext it was built with.
identifyGate An IdentifyGate (acquire(shardId): Promise<void>, release(shardId): void) when context.coordinatedIdentify is true, otherwise undefined. acquire() resolves when the shard may IDENTIFY and rejects when the parent denies it, the parent goes away, or the worker is closed. Call release() once the IDENTIFY was sent.
ready(shardIds?) Tells the parent the cluster is ready. Defaults to all of context.shardIds. The parent marks the cluster ready on any ready frame whose shards it owns, so call it once all of them are ready.
reportHealth(report) Pushes a WorkerReport ({ shards: ShardHealth[], metrics?, reportedAt }) to the parent without being asked.
onRequest(type, handler) Answers requestCluster() requests of type. The handler gets data; its return value (awaited) or thrown error message goes back. Returns the worker. Throws a TypeError for a blank type or a non-function. A request with no handler is answered with No handler for "<type>".
close() Stops listening, rejects pending IDENTIFY requests, and turns off exitOnParentLoss.

Request types starting with __lunibee: are reserved: ShardManager registers the built-in health and shard-assignment handlers under them.

import { Gateway, GatewayIntentBits, ClusterWorker, readClusterContext } from "lunibee";
const token = process.env.DISCORD_TOKEN!;
const intents = [GatewayIntentBits.Guilds];
const context = readClusterContext()!;
const worker = ClusterWorker.current()!;
for (const shardId of context.shardIds) {
const gateway = new Gateway({ token, intents, shardId, shardCount: context.shardCount });
await worker.identifyGate?.acquire(shardId);
try {
const identified = new Promise<void>((resolve) =>
gateway.on("stateChange", (change) => {
const { next } = change as { next: string };
if (next !== "CONNECT" && next !== "HELLO") resolve();
}),
);
await gateway.connect();
await identified; // release only once IDENTIFY went out
} finally {
worker.identifyGate?.release(shardId);
}
}
worker.ready();

Gateways created outside ShardManager and without the gate are not paced by the parent.

The pure functions the manager uses, exported for custom orchestrators and tests.

allocateShards(shardIds: readonly number[], clusterCount: number): number[][]

Splits shard IDs into contiguous blocks, in order; when they do not divide evenly the first clusters get one more. The same input always gives the same output. Lists at the end are empty when there are more clusters than shards. Throws a RangeError unless clusterCount is a positive integer. Before 0.4.0 the split was round-robin.

allocateShards([0, 1, 2, 3, 4, 5], 4); // [[0, 1], [2, 3], [4], [5]]
planClusters({ shardIds, clusterCount?, shardsPerCluster?, defaultClusterCount }): ClusterPlan

What spawn() does with the options, without starting anything: clusterCount splits the shards with allocateShards, shardsPerCluster fills clusters of that size (the last may be partial), and neither uses defaultClusterCount. Never more clusters than shards, and no cluster is empty. Returns { clusters, source, requested }. Throws a RangeError for both options at once, a count that is not a positive integer, or no shards.

planClusters({ shardIds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], shardsPerCluster: 4, defaultClusterCount: 1 }).clusters;
// [[0, 1, 2, 3], [4, 5, 6, 7], [8, 9]]
assertCompleteAllocation(chunks: readonly (readonly number[])[], expected: readonly number[]): void

Throws an Error naming the duplicate, missing and unexpected shard IDs unless the chunks cover expected exactly once.

validateShardIds(shardIds: readonly number[], total: number): void

Throws a RangeError unless total is a positive integer and every ID is a unique integer in [0, total).

remote turns the manager into a coordinator for ClusterHost agents on other machines. Clusters on a host are supervised, routed, paced and health-checked like local ones. Options, placement, authentication and fencing are on the ClusterHost page.