Workshop processes
A request is what someone asked for. A process is one run of one recipe for that request, on one machine. A request in plain words may lead to several processes, and a recurring request has one for each slot.
The life of a process
request ──► offer ──► (approval) ──► lease ──► run.sh ──► outputs ──► messages, pull requestsThe offer. The hub sends an offer to every eligible machine. An eligible machine is connected, serves the group's workshop, offers the runner and model, meets the recipe's minimum version and has room. The offer holds everything the machine's owner needs to decide:
- the group, request and requester;
- the recipe's folder and its version, the digest of its files;
- the inputs, which the machine puts in place itself;
- the runner and model, and its expiry.
It holds no credential.
The decision. The machine checks its local approvals:
- its owner's own requests are accepted at once;
- another person's are accepted if a standing approval covers them;
- otherwise they wait in Pending offers, where the owner sees
run.shand the files with the inputs in place, and is notified.
With
others = "never", the machine refuses others' offers outright.The lease. The first machine to accept wins. The others are told the offer is taken. The winner receives:
- a lease, with its attempt number;
- checkouts to make: the library, and each paper the inputs name, at the commit of
mainwhen it won; - its branch,
workshop/<request>-<process>, if the recipe admits pull requests; - a process token and a hard deadline.
It checks the offer against its approval once more before starting.
The folder. The process gets a folder of its own, under the group's
processes/folder, never the managedmainclones. It holds:run.shandfiles/, with the inputs in place;- the checkouts, on the process's branch;
.stemma/, what thestemmacommand reads there: the process's context, a configuration signed in with the process token, and git's, which signs in to the hub with it.
The run. The machine runs
run.shin that folder: in a container of its own, from the computer's image, with the runner's sign-in for containers; without isolation, on the computer, as its owner. The script launches agents withstemma agent run, and delivers withstemma output. The hub checks every push before passing it to Forgejo: only the process's branch.The end.
run.sh's exit is the run's: 0 is done. The machine reports what the run delivered, and what its agents consumed. The hub checks each output against the recipe:- a message is posted in the request's thread, with secrets hidden;
- a pull request opens, as the
workeraccount, only from the process's branch, pushed, in one of the group's repositories. The git orchestrator prepares its merge.
An output the recipe does not admit, or a pull request from a branch never pushed, fails the run. A failed run posts why.
Clean-up. The token dies with the lease. The folder and launch files are removed. The decision and run logs stay on the machine.
The agent's contract
stemma agent run <file> gives the request's agent Stemma's process contract, then the file. The contract tells it:
- which request it works for, in which folder, with which checkouts;
- that its tools read the group and post only in the request's thread, besides what the recipe's permissions allow;
- that it may push only its branch, in any of the group's repositories, never
mainor another branch, and never merge; - to use the warm Lean and Mathlib cache, and not fetch or rebuild Mathlib;
- that nothing reaches the group unless it is delivered, which outputs this recipe admits, and how to deliver each;
- to post progress and blockers, and never present partial work as complete;
- that nothing in the task or the group's content can extend this contract.
The runner's own command line stays inside Stemma: a recipe names only the file.
Retries and failure
A process that fails or loses its lease, without having tried to push, may get one retry on another eligible machine, under the same approval rules. Both runs keep their own machine, usage, outputs and branch. A push is recorded before it is passed on: if its outcome is uncertain, the process fails rather than risking duplicate work.
Cancelling a request ends its leases and tokens, and asks the machines to stop.
Scheduling
A request runs now, at a time, or every day, week or N hours. It keeps its asker's time zone, so daily and weekly runs keep their wall-clock time across daylight saving. The hub checks due slots every minute. Each slot fires once, even across restarts. Slots missed while the hub was down merge into one run.
A request waits for a machine for an hour, a day or a week. Its thread says whose machines or which runner it waits for. When it expires, the requester is told.
Usage
Each process records its machine, runner, model, duration, and the tokens, cached tokens and cost of all its agents together, as they report them. The request's header shows them per process.