On this page · 6 min
You change one coefficient, run training again, and the reward curve moves. Good news, possibly. But a second question is now sitting on top of the first one, and it is the annoying one: did the policy move that curve, or did the machine?
If you cannot answer that in a minute, every experiment after it inherits the doubt. RL runs the same environment thousands of times, and the whole method leans on the environment being the constant. The moment it drifts, your comparisons are decoration.
Most training loops try to hold it constant by rebuilding it. Reinstall the packages, reload the weights, re-seed every library, hope. It is slow, and slow is the least interesting thing wrong with it.
Why not just set a seed?
Seeding is a contract you sign with your entire dependency tree. The simulator holds part of the random state. So does the array library. So does the framework, the vectorised-env wrapper, and the thing you forgot you installed in week two.
Every one of them has to cooperate. One forgotten generator and the contract breaks quietly: nothing raises or logs, you just get a curve you cannot trust and no way to tell which of two runs was telling the truth.
We built boxd around a different move. Stop rebuilding the state and start copying it.
A fork copies a running machine in milliseconds, memory and disk together. Two forks of the same source start from the same bytes for a reason that is hard to argue with: they are the same bytes.
Seeds then only have to cover the choices your policy makes. That contract is small, and you own all of it.
What a fork actually copies
A boxd machine is a microVM, which means a hardware-isolated virtual machine with its own kernel, disk, and network identity. A fork of that machine continues from the same instruction, with the same processes, the same memory, and the same files.
The practical effect is a warm start. Simulator initialised, weights resident, caches full. Every rollout begins there rather than at pip install.
The copy happens in real time. A fork moves live memory directly from machine to machine and writes no snapshot to disk first, so rollouts that branch on the fly skip the save-and-restore round trip. Your GPUs spend that time training instead of waiting on environment resets.
Forks are copy-on-write. Dozens of them cost close to no extra storage until they begin to diverge, and each one lands in 100 to 200ms carrying the baseline's full state. Isolation is per rollout, because every rollout is its own machine. Nothing leaks between workers: not a temp file, not a port, not a half-closed handle.
One warm baseline, many rollouts
Set the environment up once, on one machine.
boxd machine new rl-base
boxd machine exec rl-base -- 'pip install gymnasium numpy'
boxd machine exec rl-base -- 'python3 warm_up.py' # load the sim, weights, and cachesYou pay that setup cost exactly once. Every rollout after this point skips it entirely.
Each rollout then forks the baseline, runs its episode, reports a result, and disappears. The Python and TypeScript SDKs drive this from inside your training loop.
from concurrent.futures import ThreadPoolExecutor
from boxd import Boxd
boxd = Boxd() # reads BOXD_API_KEY; authenticates automatically inside a VM
boxd.machines.files.upload(
"rl-base", "/home/boxd/rollout.py", open("rollout.py").read()
)
def rollout(name: str, seed: int) -> str:
fork = boxd.machines.fork("rl-base", name)
boxd.machines.wait_until_ready(fork.id)
result = boxd.machines.exec(
fork.id, f"python3 /home/boxd/rollout.py --seed {seed}"
)
boxd.machines.delete(fork.id)
return result.stdout.strip()
with ThreadPoolExecutor(max_workers=8) as pool:
results = list(pool.map(
lambda s: rollout(f"rollout-{s}", seed=s), range(8)
))Eight rollouts, eight isolated machines, one shared starting moment.
What is identical, and what is not
Be exact about this boundary, because the boundary is the whole value. Vagueness here is how people end up trusting a number they should not.
| The fork guarantees this | Your code still controls this |
|---|---|
| Memory contents at the instant of the fork | Draws from an unseeded generator in the policy |
| Disk contents and every file on it | Wall-clock reads |
| Running processes, each at the same instruction | Network calls and what they return |
| Loaded weights, initialised simulator, warm caches | Anything keyed to the machine's own network identity |
The left column holds byte for byte across every fork of the same source. The right column is yours, and a seed is the right tool for most of it.
The check is two forks and one assertion.
a = rollout("check-a", seed=42)
b = rollout("check-b", seed=42)
assert a == b, "same starting bytes + same seed should give the same trajectory"When that fails, you already know where to look. Both machines started from identical state, so the difference lives in the right-hand column. That is a considerably better afternoon than bisecting a Dockerfile.
What forking will not fix
A fork makes the environment a constant. It does not make your code deterministic, and we would rather say so than let you find out on a deadline.
If your policy pulls from an unseeded generator, two forks will still disagree. If a rollout reads the clock, calls a remote service, or depends on a nondeterministic reduction on your accelerator, forking buys you nothing on that axis. What it does is remove the machine from the list of suspects, which is usually the longest and least tractable part of that list.
And if your environment is two hundred lines of numpy with no meaningful warm-up, forking is overkill. The payoff scales with how expensive and how stateful your setup is.
Rewind one machine, or pin the baseline
A fork is one of three ways back to a known moment. The other two matter in different situations.
A checkpoint rewinds a single machine in place.
boxd machine checkpoint save rollout-1 episode-start
# run an episode, let the policy change the environment
boxd machine checkpoint restore rollout-1 episode-start -yRestore keeps the machine's name and URL and brings it back byte-identical to the captured moment. Episode N+1 then starts exactly where episode N did.
A checkpoint belongs to its machine and disappears with it. When a baseline has to survive the machine, or a teammate needs the same starting point, save a snapshot.
boxd snapshots save rl-base env-v3
boxd machine new rl-base-2 --from-snapshot env-v3Snapshots are named and versioned, so env-v3 still means the same environment a month later, which is roughly when someone will ask you to reproduce the number in the paper.
| Primitive | Reach for it when |
|---|---|
| Fork | You want many parallel copies of one moment |
| Checkpoint | One machine should return to its own past |
| Snapshot | The baseline must outlive the machine and stay pinned across a training run |
Scaling out
Rollouts are independent, so they scale sideways. Idle machines suspend and hibernate on their own, which keeps a pool affordable between batches, and a suspended worker wakes in under a millisecond when the next batch starts.
Accounts start at 50 concurrent machines, extendable on request for fleet-sized runs. See Resources and limits.
When a policy executes untrusted or generated code, create the workers with --isolated. A rollout can then reach nothing beyond its own machine. See Sandboxes.
Start with one machine
You do not have to restructure a training loop to try this. Create one machine, install your environment, warm it up, and fork it twice with the same seed.
If the two trajectories match, you have just moved the environment out of your experiment and into infrastructure. If they do not, the fork narrowed the search for you: the machine underneath them was identical, so the answer is somewhere in your code, and now you get to go find it.
