On this page · 12 min
Deep dive is a highly technical series where I go deeper into the technology that makes boxd tick. This is not for the faint of heart!
We run virtual machines for a living. Not the big kind you SSH into once and then forget about for three years, and not the throwaway kind that gets destroyed at the end of a CI job. The middle kind: a real Linux machine that starts in milliseconds, keeps its files and its running processes between sessions, and goes to sleep when nobody is looking at it.
Mostly people use them as remote development environments. You SSH in, your editor connects, your shell history is where you left it, and the coding agent you started last night is still going. But a machine is a machine, so plenty of them end up running something small and permanent as well: an internal tool, a website, a scheduled automation, a bot.
To run those machines we have our own VMM. It has evolved a lot over time, and what matters for this post is that we own both ends of the boundary: the thing running the VM, and the kernel running inside it.
The most important rule we have is this one:
We do not trust the guest.
It is somebody else's code, running as somebody else's user, doing somebody else's questionable things, on a machine we are renting to a stranger. Everything on our side of the boundary is written as if the guest is actively trying to lie to us, because sooner or later one of them is.
And yet. We also want to know what is going on in there. Not "how much CPU is it using", that is easy and boring. Actual application-level facts. Is it up? Is it ready? Is it serving anything?
Hold onto that, because there is a third side to it. We do not trust the guest, we do want to know what the guest is doing, and we are not willing to rummage around inside somebody's machine to find out. It is their machine. Their files, their processes, their business. Zero trust going in, real visibility coming out, and no snooping in between.
The public URL
Every machine gets a URL. You start a machine, you SSH in, you run whatever you want, and if it is listening on a port then https://myapp.boxd.sh serves it. No load balancer to configure, no ingress YAML, no listener rules, nothing to declare in advance.
The first two thirds of that are unremarkable. Our own authoritative DNS answers for the zone, so myapp.boxd.sh resolves to one of our proxies. The proxy terminates TLS, reads the SNI, and looks the name up in the cluster state to find out which machine that is and what IP it has.
Fine. Solved problem. Everyone does this.
Now the proxy is holding a decrypted HTTP request, it knows exactly which machine it belongs to, it has a route to that machine, and there is exactly one thing standing between it and forwarding the request:
Which port?
The boring answers
There are three normal ways to answer that, and I want to walk through them because the point is not that they are bad. The point is that they are all the same answer wearing different hats.
Ask the user. A --port flag, a field in the dashboard, an EXPOSE line if you are a container platform. This is what everyone does and it works great. It is also one more thing to declare, one more thing to get wrong, and one more thing that silently disagrees with what your app actually does at 2am.
It has a worse problem here, though, and it is specific to what our machines are. A machine is not a deployment. You start one, and then an hour later you SSH in and run a dev server, and then tomorrow you swap it for a different one on a different port. Nothing declared at create time can possibly know about that. The question is not answered once, it is answered continuously, by whatever you happen to be running right now.
Poll it. Sit outside the machine and hammer connect() at it until something stops refusing. This works in the sense that a smoke detector you test by lighting a fire works. Every answer is stale by the time you have it, your interval is either too slow for the user or too expensive for you, and, critically: poll what? You need a port to poll. You are back at the previous answer, just with more sockets.
Look inside. Read /proc/net/tcp in the guest, or run a little agent in there that reports back what it sees. Now go back and read both rules again. Anything the guest hands us is a claim rather than a fact, and going and reading the guest's process table for ourselves is exactly the kind of rummaging we said we were not going to do.
So we did the fourth thing, which is not really a thing, which is that we patched the kernel that runs inside the machine.
Thirty-one lines in net/socket.c
Here is __sys_listen, from the guest kernel we ship. The parts that are not ours are exactly as upstream left them:
int __sys_listen(int fd, int backlog)
{
struct socket *sock;
int err, fput_needed;
int somaxconn;
bool compatible_family_trigger = false;
char trigger_data[7] = {0};
sock = sockfd_lookup_light(fd, &err, &fput_needed);
if (sock) {
if (sock->sk) {
compatible_family_trigger = true;
// trigger_data[0..1] = port
trigger_data[0] = sock->sk->sk_num & 0xFF;
trigger_data[1] = sock->sk->sk_num >> 8;
// trigger_data[2..5] = address
trigger_data[2] = sock->sk->sk_rcv_saddr >> 24;
trigger_data[3] = sock->sk->sk_rcv_saddr >> 16;
trigger_data[4] = sock->sk->sk_rcv_saddr >> 8;
trigger_data[5] = sock->sk->sk_rcv_saddr;
boxd_sys_trigger(BOXD_SYS_LISTEN_BEFORE, trigger_data);
}
somaxconn = READ_ONCE(sock_net(sock->sk)->core.sysctl_somaxconn);
if ((unsigned int)backlog > somaxconn)
backlog = somaxconn;
err = security_socket_listen(sock, backlog);
if (!err)
err = sock->ops->listen(sock, backlog);
fput_light(sock->file, fput_needed);
}
if (err == 0 && compatible_family_trigger) {
boxd_sys_trigger(BOXD_SYS_LISTEN_AFTER, trigger_data);
}
return err;
}That is the whole feature. Your app calls listen(), like every server that has ever existed, and on the way through the kernel it tells on itself.
Two triggers, not one. _BEFORE fires while we still only know the intent, before the LSM hook, before sock->ops->listen. _AFTER only fires on err == 0, so it means it actually happened. The syscall runs at full speed in both cases, but the ordering means there is never a window where the app is listening and we have not heard about it yet. There is no race to lose, because we are told first and confirmed second.
bind() gets the same treatment, reading the port and address out of the sockaddr_in instead of the socket. The app does not know, does not care, and did not have to be recompiled, rewritten or configured. Nothing about the app changed. We changed the floor it is standing on.
The wire is a port. Just a port.
Here is how the trigger gets out:
typedef struct {
unsigned char code;
// 7 bytes data
unsigned char data[7];
} boxd_sys_trigger_data;
// Ensure the size of the struct is 8 bytes
typedef char boxd_sys_trigger_data_incomplete_size[sizeof(boxd_sys_trigger_data) == 8 ? 1 : -1];
void boxd_sys_trigger(unsigned char code, char data[7])
{
boxd_sys_trigger_data trigger_data;
trigger_data.code = code;
if (data) {
memcpy(trigger_data.data, data, sizeof(trigger_data.data));
} else {
memset(trigger_data.data, 0, sizeof(trigger_data.data));
}
/* Send 8-byte trigger as two 4-byte PIO writes to port 0x510 and 0x514.
* The VMM reconstructs the 8 bytes from these two writes. */
outl(*(u32 *)&trigger_data, BOXD_TRIGGER_PIO_PORT);
outl(*((u32 *)&trigger_data + 1), BOXD_TRIGGER_PIO_PORT + 4);
}One byte of opcode, seven bytes of payload, and a compile-time assertion that it stays exactly eight bytes forever. That char array[cond ? 1 : -1] trick is how you did static_assert before static_assert, and it is here because the day this struct silently grows to twelve bytes is the day the other end starts reading garbage and nobody finds out for a month.
Then two outl instructions to port 0x510, and we are out of the guest.
That is the entire transport. No virtio device. No driver to probe. No shared memory ring, no descriptor table, no negotiation, no MMIO region to map. An out instruction on x86 is a VM exit, which means the moment it retires, the vCPU thread is sitting in our VMM with the bytes in hand. It is the oldest, dumbest IPC mechanism available on the platform, and it is perfect for this, because it works from any context at any point in boot with no setup at all. The subsystem comes up at subsys_initcall and can start snitching immediately.
I would like to say this design was the result of careful evaluation. It was mostly the result of wanting to be done by dinner.
The other end
On the VMM side, the guest manager device is registered on the PIO bus at 0x510, and reassembles the eight bytes out of the two writes:
// Port 0x510 - trigger data, low 4 bytes
0 => {
let len = data.len().min(4);
self.trigger_buf[..len].copy_from_slice(&data[..len]);
self.trigger_pos = len;
}
// Port 0x514 - high 4 bytes, completes the trigger
4 => {
let len = data.len().min(4);
self.trigger_buf[4..4 + len].copy_from_slice(&data[..len]);
self.trigger_pos += len;
if self.trigger_pos >= 8 {
self.process_trigger();
}
}process_trigger matches on the opcode byte, pulls the port and address back out of the payload, and turns the whole thing into a PortListen event. From there it is ordinary asynchronous plumbing.
The problem is that it will not shut up
Here is what I did not expect. The hard part was never getting the information out. The hard part is that once the kernel starts telling you about every listen() call, you find out how many of them there are.
A running machine is a much noisier place than the mental model suggests. The language runtime opens a debug or inspector port. The framework opens a second socket for hot reload. Something binds a metrics endpoint nobody asked for. There is a local resolver, a supervisor control socket, a health check listener, an IPC socket that happens to be TCP because that was easier for whoever wrote it. Every one of those is a listen(), every one of those is faithfully reported to us in whatever order the app happens to start them, and not one of them is labelled "this is the one the user meant".
So we got perfect information, and immediately discovered that perfect information is not the same thing as the answer.
The filter has two halves:
const WELL_KNOWN_PORTS: [u16; 6] = [80, 443, 8080, 8000, 3000, 5173];
GuestEvent::PortListen { port, addr } => {
let bindable = addr.is_unspecified() || addr.octets()[0] == 127;
if let Some(priority) = WELL_KNOWN_PORTS.iter().position(|&p| p == port)
&& bindable
{
let should_update = match best_port {
Some((current_priority, _)) => priority < current_priority,
None => true,
};
if should_update {
best_port = Some((priority, port));
reporter.push_port_detected(&vm_id, port as i32).await;
}
}
}The address half throws out anything bound to one specific interface address, on the theory that a service which picked an exact address knows something about its network that we do not, and is talking to something other than the internet.
The port half is a beauty contest. Position in the array is the priority, lower wins, and should_update only fires on a strict improvement. That last part matters more than it looks: it means the app can keep opening sockets forever and never demote itself. Once something has claimed 80, no amount of subsequent chatter takes the URL away from it.
And yes, 5173 is in there because Vite.
I have made peace with this. The list is not science. It is a ranking of what a person would guess if you showed them an ss -tlnp dump and asked which one is the website. The kernel patch is the clever half, and it produces a firehose. The six-element array is the dumb half, and it is the reason the feature feels like magic instead of feeling like a log file.
About that trust thing
Let us go back to the rules, because a careful reader is already ahead of me here.
That PIO port is not privileged. It is sitting there at 0x510, and any process in the guest that can get ioperm can write to it. I know this for a fact, because our own init process does exactly that:
unsafe { libc::ioperm(0x510, 1, 1) };
// ...
std::arch::asm!("out dx, al", in("dx") 0x510u16, in("al") trigger_byte);So the guest can lie to us. It can claim it is listening on 8080 when it is not. It can announce ports it never opened, in any order, all day long.
That is fine, and the reason it is fine is the actual design principle behind the whole thing: the trigger is a hint, not a capability.
Nothing in that message authorizes anything. It cannot name another tenant's machine, because the VMM already knows which machine it is talking to and the guest never gets a say in that. It cannot open a port, cannot punch through a firewall, cannot make the proxy route traffic anywhere it was not already willing to route traffic for that exact machine. The worst outcome available to a guest that lies to us is that its own URL points at its own wrong port, and it serves itself a connection refused.
It is also the narrowest possible thing to know. We are not reading your files. We are not walking your process table, or watching your syscalls, or shipping your logs anywhere. One number, volunteered by the kernel at the moment it stops being a secret anyway, because a port you are listening on is a port you are inviting the world to connect to.
That is the rule we ended up with, and it generalizes further than I expected when we wrote it:
Never let the guest tell you something you would otherwise have to trust. Let it tell you something you would otherwise have to guess.
Guessing is what polling was. Guessing is what the config field was. Replacing a guess with a hint costs nothing when the hint is wrong, and buys an enormous amount when it is right, which is almost always, because the overwhelming majority of guests are not attackers. They are a dev server that just wants somebody to notice it came up on port 3000.
Where the port ends up
The detected port travels from the VMM into the machine's row in our cluster state, as detected_port, and only for machines left in automatic port mode. It is replicated through Raft, so every proxy in the cluster knows it.
And it is copied on snapshot and on fork, which is my favourite detail in the whole system. When we fork a running machine to make a new one, the child inherits its parent's answer instead of having to call listen() again to rediscover a fact that was already true.
One syscall, eight bytes, one VM exit, propagated through consensus, surviving reproduction.
This was only the first thing we taught it
listen() and bind() were where we started, because the public URL is the feature people notice. They are not where we stopped. The same eight bytes now carry userspace-ready signals, exit codes and file events, because once you have a wire out of the guest, everything starts looking like something the guest ought to be telling you about.
And then there is what happens when you fork a live machine, and the child wakes up on a host whose clock is lying to it, with a dead APIC timer, not yet knowing its own IP address. That one needs an NMI.
