On this page · 8 min
Deep dive is a highly technical series where I go deeper into the technology that makes boxd tick. This is not for the faint of heart!
We fork running machines. Not "boot a second copy from the same image": take a machine that is up, with your editor attached and your process tree running, and produce a second one that is byte-identical and running, in about a tenth of a second. The child does not boot. It resumes, holding everything the parent was holding at the moment we paused it.
That is the feature. This post is about the few hundred microseconds afterwards, when the child has to work out that it is not its parent.
Because a perfect copy is precisely the problem. The child comes up believing every single thing the parent believed, and half of those things are now false. It has the parent's IP address, which belongs to a machine that is still running and still using it. It has the parent's hostname. It has the parent's entropy pool, which is worse than it sounds. Its clock is calibrated against a different physical CPU. And on AMD, its timer is dead in a way that looks, from inside, exactly like a timer that works.
None of that is fixable from outside. Some of it is only knowable from inside.
Nobody is home
Here is the constraint that shapes everything else.
When we restore a VM, every vCPU comes back halted. There is no thread running in the guest. No process to signal, no interrupt handler waiting, no daemon polling a socket. The guest is a complete, consistent, motionless snapshot of a machine, and it stays that way until something makes it execute an instruction.
So we cannot ask it to fix itself, because there is nobody in there to ask. And we cannot fix it from the host, because the things that are wrong are guest kernel state: the clock page, the timer, the network configuration, the hostname. The host can write the guest's memory, which it does, but it cannot make the guest notice.
What we need is a way to make a stopped guest run one specific piece of code, before it runs anything else, with no cooperation from anything already scheduled in it. That is a non-maskable interrupt.
So the VMM writes a small structure into guest physical memory, injects an NMI, and only then starts the vCPUs. The guest kernel carries a handler for it, and the handler does almost nothing:
phase = *(volatile u8 *)((char *)fork_mailbox_page + FORK_PHASE_OFFSET);
if (phase != FORK_PHASE_REQ_RESUME)
return NMI_DONE;
irq_work_queue(this_cpu_ptr(&fork_resume_work));
return NMI_HANDLED;NMI context is the most hostile place to run code in a kernel: you can arrive in the middle of anything, including code holding the locks you would want. So it reads one byte to decide whether this NMI is ours, queues work for normal context, and gets out. Every vCPU takes the NMI, and every vCPU queues its own, because what is broken is per-CPU.
The byte it reads sits in a page the host filled in first, opening with an eight-byte magic, because a page of guest physical memory is a terrible API and the only thing worse than reading a stale mailbox is not noticing you did. Behind the magic is everything the child needs to know about itself. It has to be readable at the exact moment the guest has nothing: no device, no driver, no filesystem, no network.
The clock is lying
The first thing that happens in normal context is that time gets fixed.
Under KVM, a paravirtualised guest does not read time off the hardware. The host publishes a small structure into a shared page (a TSC offset, a scale factor, a multiplier) and the guest computes time from its own TSC using those numbers. Fast, and correct exactly as long as the numbers describe the CPU you are actually running on.
A forked guest wakes up holding its parent's copy of that page. Same fields, different physical CPU, and nothing in the process tells it to recalculate. From inside, nothing is obviously wrong. Time simply comes out incorrect, and every derived quantity comes out incorrect with it. The way we found this was a guest computing timer periods of 992 nanoseconds and behaving exactly as unwell as that implies.
The fix is three lines:
rdmsrl(0x4b564d01, msr_val); /* MSR_KVM_SYSTEM_TIME_NEW */
if (msr_val)
wrmsrl(0x4b564d01, msr_val);Read a register, write the identical value back. A no-op by every reasonable definition, and it fixes the problem completely, because the value was never the point: the write traps to KVM, and KVM recomputes the clock parameters for this VM against this host's TSC. We are not setting anything. We are ringing a bell.
The timer is dead and looks fine
Restoring a VM restores its local APIC, which faithfully puts back every register the parent had. On AMD, that is all it does. The host-side timer that actually makes the guest's clock tick fire is not created by restoring registers. It is created when the guest writes the initial count register, and a restored guest already did that before the snapshot, so it never does it again.
The result is a timer that is perfectly configured and completely inert. Read the registers from inside and they say exactly what they should say. Nothing fires. The scheduler's tick never arrives.
apic_write(APIC_TMICT, 23700000);Same shape of trick as the clock: we write the count register to cause the exit that makes the host build the timer behind it. The value matters much less than the fact that writing it is observable to the host.
This is the kind of bug that is only findable from the guest and only explicable from the host, and it took an embarrassing amount of time, because every diagnostic you would reach for reads a register and tells you the timer is fine.
Who am I, and the part that actually matters
Time and timers are fixed by the kernel in interrupt context, in microseconds. Identity needs userspace, so it takes a different route: one CPU schedules a workqueue, which sends a real-time signal to PID 1. Our init has been sitting in sigwaitinfo since before it spawned a thread, with the mailbox already mapped, so it can read the child's new identity the moment the signal lands without asking anyone for it.
The first thing it does with that mailbox, before the network and before the hostname, is reseed the kernel's random pool from 64 bytes the host generated while writing the page.
Here is why that is first.
Fork a machine twice and you have two children whose kernel random pools are bit-for-bit identical, because the pool is memory and we copied the memory. They will produce the same "random" numbers, in the same order, until something reseeds them. Every session token, every nonce, every ephemeral key, every UUID that any process in either machine generates from that point is a value the other machine will also generate.
That is not a performance bug or a correctness curiosity. That is the fork feature quietly handing you a security incident, and it is invisible in every test that does not specifically look for it, because each machine's numbers look perfectly random on their own.
So the host reads 64 bytes from its own /dev/urandom per child, and the guest mixes them in and credits them before it is allowed to do anything else. It costs one read on the host and microseconds in the guest, and it is the highest-value line in the whole sequence.
The budget
What is left is prosaic: set the IP, the gateway, the netmask, the nameserver, rewrite the hostname.
The first version shelled out to ip. It worked, and it cost between 300 and 700 milliseconds on a fork whose entire budget is under 200. Nothing to do with ip itself: in a freshly restored guest the page cache is cold, so every invocation faults in the binary, then libc, then the loader, from disk. Do that four times and the cost of finding the programs dwarfs the work.
So it is raw netlink now. Sub-millisecond syscalls instead of hundreds of milliseconds of cold start, for the same operations. We simply stopped paying to start a program in a machine that has just woken up and has not got its filesystem cache back.
What the child never finds out
Step back and the shape of it is odd in a nice way. A forked machine is repaired by a sequence that starts with an interrupt that cannot be masked, proceeds through two register writes whose values are irrelevant, and ends with a real-time signal to PID 1. Four different mechanisms, because there are four different kinds of "the guest is not listening" to get past.
And the process inside the machine, the one you actually care about, sees none of it. Its clock never jumped. Its socket never dropped. It was paused between two instructions and resumed between the same two instructions, on a different machine, with a different address and a different pool of randomness, and it was never told.
That is the part I find genuinely strange about this feature, and I have stopped expecting to get used to it.
Next: how the copy gets made in the first place, which needed a system call that Linux does not have.
