Command line¶

The launcher takes one verb and that verb's flags. The verbs are train, serve, validate, version and the internal agent.

launch train [-config launch.yaml] [-print]
launch validate [-config launch.yaml]
launch version

An unknown verb or flag, a positional argument or a flag given twice is refused with exit 64.

launch train¶

Run it from the batch step. It resolves the resume point once, chooses the rendezvous port, starts one launch agent per node with srun, merges their reports into the cause document and exits with the merged outcome. It needs srun and the agents' torchrun on PATH.

While it runs it replaces $DISPATCH_CAUSE_FILE.beat every 30 seconds with a launch.beat.v1 beat: seq rises on every write, nodes counts the nodes heard in the last minute, and step and phase are the highest step any node last reported from the payload's heartbeat file, and that node's phase. Nodes report every 10 seconds and the file is replaced every 30, so what you read can trail the heartbeat by about 40 seconds. Before any heartbeat, or with no heartbeat section, they are 0 and train. A failed write is logged and doesn't stop the run.

Inside a Kubeflow pod, where PET_NODE_RANK is set, it runs as that pod's agent. There is no coordinator: each pod writes its own $DISPATCH_CAUSE_FILE, and node rank 0 prints LAUNCH-BEAT and the beat to stdout every 30 seconds.

With a checkpoint section, node rank 0 runs publish resolve once and serves the answer over HTTP on every interface, on rendezvous.resumePort, by default PET_MASTER_PORT plus 1. The other nodes fetch it from PET_MASTER_ADDR, so every node resumes from the same save. torchrun starts on node rank 0 once every other node has fetched, or at rendezvous.joinTimeout. The listener is closed by then, but the connections it answered stay in TIME_WAIT for a while, so a payload given this port has to bind it with SO_REUSEADDR. The other nodes wait up to rendezvous.joinTimeout, which also has to cover node rank 0's resolve. They exit 70 if it passes, or if the answer is for another attempt. Node rank 0 exits 70 if its port is already in use, or if its resolve outlasts rendezvous.joinTimeout, since the other nodes have stopped by then. A resolve on a one-node job has no such bound: on a POSIX root, only a signal ends a hung one. The answer is unauthenticated. A URL location with userinfo, a query string or a fragment is refused before it is served, but a location with no scheme isn't checked, so a credential in a plain path would be served. The helper defaults to /opt/dispatch/publish, and PET_NNODES must be set.

launch train -print¶

Prints as JSON what a run would do on this node, and starts nothing it would run: no torchrun and no publish helper. It reads the same file, makes the same refusals and exits 64 for them.

• -config works as it does for validate.
• Under Slurm it runs srun --version, because the grace depends on which release this is.
• torchrun is the argv node rank 0 would use, and on Slurm a rendezvous port of 0 with portChosenAtStart means the port is chosen when the job starts.
• resumePoint is unresolved with a checkpoint section, since resolving it runs the helper, and none without one.
• controlDir is launch-<attempt>-<node rank> under $TMPDIR, or /tmp without it. Each rank's control files need it to be on the node's own disk, and $TMPDIR doesn't guarantee that, so check it on a site that points it at shared storage.
• gpu is the GPU state the run's document reports: off, not-built, absent or nvml. It's absent where the launch-nvml build can't start NVML or NVML shows no GPUs, and the line before it says why. A run can still report absent if the first reading fails or finds every GPU lost or barred to the process, which -print doesn't read. A GPU that's already lost when some others still read isn't reported at all: it's absent from the first reading, so nothing shows it went missing, and it has no UUID to name. Under Slurm this is what the node you ran it on sees, and each node's agent decides for itself.

The output is safe to log only with these limits:

• Every value from train.env prints as ***. The launcher's own LAUNCH_* values print with go/redact's masking, so a credential in a checkpoint URI is removed.
• In train.command, a value prints as *** when its name looks like a secret's (it contains token, secret, passw, pwd, key, cred or auth, in any case). That covers --token x, --token -x, --token=x, TOKEN=x, --opt=token=x, --token:x and Authorization:x. A flag with such a name and no = also hides the whole of the next argument, even if that was another flag.
• An argument that holds such a name and also a space prints as *** whole, as with sh -c "... --token x". go/redact's URL credentials and token shapes are also removed.
• Any other argument prints as written. A secret under an innocent name such as --name x or name:x, or with no name at all, is printed, so pass secrets through train.env.

launch agent¶

The per-node half of a Slurm run, which the coordinator starts over srun. Don't run it yourself. It runs torchrun, watches it, and sends its report to the coordinator over stdout.

It makes its control directory, $TMPDIR/launch-<attempt>-<node rank>, and refuses if that already exists, because a leftover error file would be read as this run's. It removes the directory when the run ends.

The helper's location reaches each agent as a flag, which ps shows, so train refuses a URL with userinfo, a query string, a fragment or a form it can't parse. It can't recognise a credential anywhere else in a location.

launch serve¶

launch serve [-config launch.yaml] [-print] [-- COMMAND...]

Starts vLLM on 127.0.0.1 from the file's serve section and waits until it's ready. With -- COMMAND, it then runs COMMAND, stops vLLM when COMMAND exits, and exits with COMMAND's code. Without one, it serves until SIGTERM, SIGINT or SIGHUP and exits 143.

vLLM is ready when GET /health answers 200, GET /v1/models lists serve.model.name and every adapter, and GET /version answers. It's polled every serve.ready.interval until serve.ready.timeout.

COMMAND receives these variables, and vLLM doesn't. serve.env goes to vLLM only.

The launcher's stdout belongs to COMMAND, so vLLM's stdout and stderr both go to the launcher's stderr.

Before starting anything, it refuses with 64 if COMMAND can't be found, the port is taken, or the manifest's directory is writable by anyone else or owned by another user.

To stop a child it sends SIGTERM, waits serve.stopGrace, then kills the child's whole process group. On a signal, COMMAND is stopped first and vLLM after it, so stopping can take twice serve.stopGrace.

If vLLM dies under COMMAND, the launcher doesn't stop COMMAND, and COMMAND's code is still what it exits with. If vLLM dies with no COMMAND, the launcher exits 70 with the last 40 lines of vLLM's stderr.

The manifest¶

The manifest, launch.serve-manifest.v1, is at $TMPDIR/launch-serve-<uid>-<port>/manifest.json, or under /tmp without $TMPDIR. It appears once vLLM is ready, a run removes any earlier one when it starts, and it stays after the launcher exits. That lets a caller tell the launcher's 69 from a COMMAND that exits 69: if the manifest isn't there, vLLM never became ready.

argv is exact and isn't redacted. The file is readable only by the user who ran the launcher, but go/assess copies it whole into its results, so treat argv as something that leaves the node. launch.yaml's serve section lists what's refused to keep credentials out of it, and what can't be.

launch serve -print¶

Prints as JSON the URL, served names, manifest path, vLLM command line and COMMAND, and starts nothing. It makes the same refusals as a run. Values from serve.env print as ***, and the command lines are hidden as launch train -print hides train.command.

launch validate¶

Reads the file and checks it, without reading the environment or starting anything.

• -config is the file to check, launch.yaml in the working directory by default. There is no environment override or search path.
• A good file prints nothing and exits 0.
• Any refusal exits 64, and each one is printed to stderr with its path in the file. launch.yaml lists what is refused.

The file itself must be a regular file (a symlink to one is fine) of at most 1 MiB, with something in it. A file holding only comments, null or {} counts as empty.

gpu.health: required is refused unless the binary was built with the nvml tag: go build -tags nvml -ldflags=-extldflags=-Wl,-z,lazy -o launch-nvml ./cmd/launch, with cgo on. Without lazy binding, a linker that binds at start stops the binary from starting at all. That binary loads libnvidia-ml.so.1 when it runs, and links dynamically, so it has to run against a glibc at least as new as the one it was built on.

launch version¶

Prints launch and the module version on one line, and exits 0. A binary built from a checkout prints (devel) or a pseudo-version.

Exit codes¶