ml-experiment control plane

A college Slurm cluster is a great place to learn why manual GPU wrangling does not scale. You get an allocation, you lose an allocation, your job dies at 3 a.m., and the model you were serving disappears with it. I built ml-experiment, a Python control plane (managed with uv) so that my own experiments stop depending on me being awake.

The core abstraction is a retained lease: acquire an allocation, pin it, and then dispatch work into it — training runs, probes, serving — without re-negotiating with the scheduler for every command. Leases span H200, RTX PRO 6000, A100, L40S, and L4 nodes, which means the control plane has to normalize a genuinely heterogeneous fleet instead of assuming one GPU class.

On top of the lease it handles model qualification (does this checkpoint fit here, and does it produce sane output before anyone depends on it), durable serving through vLLM and llama.cpp/GGUF including speculative decoding, plus the telemetry and diagnostics that make a remote job debuggable: what is running, where, for how long, and what it last said.

The part I am most confident in is the 307 automated tests. Cluster tooling is exactly the software people refuse to test — the environment is awkward, the failure modes are slow, and the temptation is to validate by hand once and move on. That is how you get a control plane that breaks silently at 2 a.m. Most of those tests are about state transitions and recovery: what happens when a lease is cancelled mid-dispatch, when a worker stops heartbeating, when a backend port is already taken.

What I'd do differently: model qualification is the piece I would write on day one. Cluster tooling is shared by nature, so the cost of a wrong answer lands on whoever else depends on the lease, not on the person who skipped the check.