How to run a self-hosted AI agent on infrastructure you control

What a self-hosted AI agent is made of

A self-hosted AI agent is one where the parts that see your data run on hardware you control. Any agent can be self-hosted this way, but "self-hosted" is only as true as its weakest layer, so it helps to name the layers first.

Take a coding agent on a developer's laptop. It reads files from your repository, sends them to a model, gets back code, and runs it in a shell. That loop touches:

• The model, the large language model (LLM) that does the thinking. Everything the agent reads goes into its prompts, so whoever runs the model sees your code.
• The machine the agent works on: its files, its shell, the code it runs and the keys it holds.
• The harness, the agent program itself (aider, OpenHands, Claude Code, your own loop), plus the state and logs it keeps.

A platform team running fifty agents has the same layers, fifty times over. Install the harness on your own server and you have self-hosted one layer out of three. If the model is still a US provider's API, every prompt crosses to them. Picking the provider's EU region decides where data is stored, but US law (18 U.S.C. 2713, added by the CLOUD Act) obliges providers under US jurisdiction to hand over data they hold wherever it is stored. For a European team, that is usually why the question comes up.

Which parts do you have to host?

Decide layer by layer, because the cost is very uneven. The harness and the machines are ordinary Linux workloads. The model is the expensive part: it needs GPUs, and many teams find the best hosted models still stronger than the open-weight ones they can run themselves, especially for coding. That is why most self-hosting guides stop at the hybrid setup.

Hybrid is fair when you've decided prompts may leave: an EU region then settles where they are stored, though not whose law reaches them. If nothing obliges you to keep prompts and code in house, a hosted service in an EU region is less work, because self-hosting means you own uptime, upgrades and backups.

Who operates the platform is a separate choice. With BYOC (bring your own cloud), a vendor runs either setup inside your cloud account, which fits a team that needs data in its own account but has no one to run a platform. boxd and E2B both offer it. The data stays in your account, but the vendor's engineers need access to operate it, which running it yourself avoids.

What you need before you start

Self-hosting isn't hard for the harness and the machines; it gets hard at the model.

• For the model: GPU servers with enough memory for the weights. A rough rule: at 16-bit precision a model needs about two bytes of GPU memory per parameter, plus room for the context. Mistral Small 4, the model in step 1, has 119 billion parameters (the "small" is relative), so it needs several GPUs in one server, while smaller or quantised models (stored at lower precision) fit on one GPU.
• For the machines: Linux hosts that can run virtual machines (KVM, the virtualisation built into the Linux kernel), either on your own servers or in a cloud account. In the cloud that means instance types with nested virtualisation, such as N2 on Google Cloud, or C7i, M7i, R7i, C8i, M8i, R8i and .metal types on AWS.
• For the boxd installer used in step 2: Docker on the computer you install from, and a DNS zone you can delegate, such as agents.example.com.

The steps below build the fully self-hosted version. For the hybrid version, skip step 1 and point the harness at your provider's API instead.

Step 1: Serve an open-weight model

Run the model behind an API on a server in your own network. vLLM is a common choice because it serves the same API shape as OpenAI, which almost every harness can talk to. The example uses Mistral Small 4, an open-weight model from a French company, released under Apache 2.0:

vllm serve mistralai/Mistral-Small-4-2603 \
  --tokenizer_mode mistral --config_format mistral --load_format mistral \
  --tensor-parallel-size 4 \
  --api-key "$LLM_TOKEN"

This downloads the weights, splits the model across four GPUs and starts an API on port 8000. The three mistral flags load the model in Mistral's own format; check the model card for any others. When the log says the server is up, curl -H "Authorization: Bearer $LLM_TOKEN" http://localhost:8000/v1/models lists the model.

Keep this server on a private network that only your agents can reach. The vLLM docs warn that --api-key only protects some endpoints, so the token is not enough on its own. For a single server, Ollama does the same job with less setup and serves the same API shape on port 11434, but it has no authentication of its own, so the private network is all that protects it.

Step 2: Give each agent its own machine

An agent runs code an LLM wrote, and anything it reads (a web page, an issue, a file) can carry instructions that steer that code, which is called prompt injection. So the agent needs a machine where a bad command damages only itself: its own isolated virtual machine with its own kernel, not a container on the server that holds your keys. With many agents you need many machines, created and deleted on demand, which is the part teams usually solve with Kubernetes.

You can run a VM per agent on your own infrastructure with boxd, E2B or Coder:

• Self-hosted boxd gives each agent a full Linux VM on KVM, on Google Cloud, AWS or bare metal. The platform is a single Rust binary, so there's no Kubernetes, external Postgres or Redis to run.
• E2B runs each sandbox as a Firecracker microVM with its own kernel. Its Apache-2.0 repository has a Terraform-based self-hosting guide for AWS, Google Cloud and Azure, so it's the one to pick if you need Azure.
• Coder runs workspaces as VMs, Kubernetes pods or Docker containers, so the isolation depends on the template you write. It documents fully air-gapped deployments and has the wider IDE support and governance features.

The rest of this guide uses boxd. These are the Google Cloud install steps; AWS and bare metal use their own example config, and the installer's docs/INSTALL.md has every sign-in command.

git clone https://github.com/boxd-sh/self-host
cd self-host
./boxd-install gcloud auth login --no-launch-browser
# a key that encrypts the cluster's state; only you hold it
( umask 077; set -C
  printf 'export BOXD_STATE_KEY=%s\n' "$(openssl rand -hex 32)" > ~/.boxd-mycluster.env )
source ~/.boxd-mycluster.env
./boxd-install gcloud storage buckets create gs://mycluster-state \
    --location europe-west4 --uniform-bucket-level-access --public-access-prevention
cp cluster.example.toml cluster.toml   # set every value marked CHANGE
./boxd-install validate
./boxd-install providers edit          # the console's sign-in (Google OAuth client)
./boxd-install apply 2>&1 | tee apply.log

The installer runs its tools inside Docker. In cluster.toml you choose the project, region and zone the whole cluster runs in, so pick an EU region (the bucket above sits in europe-west4, in the Netherlands) to keep the machines and their disks there too. providers edit stores the OAuth client you create for console sign-in, validate checks the config, and apply builds the cluster. Back up the state key and the bucket, because the cluster's identity depends on them.

When apply finishes it prints the name server records to add at your DNS provider. Once those resolve, open https://app.<your zone> and sign in straight away: the first person to sign in owns the cluster's org, so don't leave a new cluster unclaimed. The console's onboarding then walks you through installing the CLI and starting a first machine.

Step 3: Limit what agent machines can reach

The model and the machines are now yours, but an agent can still send data out over the network, so set this up before any agent runs:

• Allow only the hosts tasks need. Add egress rules where agent traffic leaves your network (the cloud firewall for the worker hosts, or your own firewall on bare metal) so agent machines reach the model server and the few hosts their tasks need, such as your Git server and package mirror, and nothing else. Without them, a prompt injection can make the agent upload your code to any server. Rules on a worker host apply to every agent on it, so keep workloads that need wider access elsewhere.
• Keep long-lived keys off agent machines. The vLLM token gives access to the model and nothing else, so it can go on agent machines. Keys to other systems, such as your Git host or cloud account, should be scoped to one task and expire with it, so a leaked key is worth little.

Step 4: Start an agent and point it at your model

With self-hosted boxd from step 2, create a machine for the agent and open a shell on it:

boxd new agent-1
boxd connect agent-1

boxd new creates the machine and boxd connect drops you into a shell on it. Install a harness there. This example uses aider, a coding agent that works with any OpenAI-compatible endpoint:

curl -LsSf https://aider.chat/install.sh | sh
export OPENAI_API_BASE=http://llm.internal:8000/v1   # your model server's address
export OPENAI_API_KEY=<the vLLM token from step 1>
aider --model openai/mistralai/Mistral-Small-4-2603

aider sends every request to your vLLM server, so the code it reads never leaves your network. Most harnesses take a base URL and token the same way; check yours does.

Once one machine has the harness installed, save it before you add any key, and start the next agents from that copy. For one-off tasks, give the machine a timer so it doesn't outlive the job:

boxd snapshots save agent-1 agent-base
boxd new agent-2 --from-snapshot agent-base
boxd new task-42 --from-snapshot agent-base --delete-after 2h

A machine started from a snapshot comes up with its processes already running. --delete-after 2h deletes it two hours later, whatever state it is in; agents can also delete their own machine when they're done. This is how coding agents run in parallel, each on its own machine, and the CLI reference has the rest.

How to choose your setup

Match the setup to what must stay in house, then to how much you want to operate:

• One developer, their own code, prompts may leave: a hosted model and the agent on a laptop or a VM you already have. You don't need a platform.
• Code and data must stay, prompts may leave: hybrid. Host the machines and the harness, keep a hosted model in an EU region.
• Nothing may leave: fully self-hosted, with an open-weight model on your own GPUs. If the network must be closed off entirely, boxd offers air-gapped installs on request, and Coder documents air-gapped deployments. A standard self-hosted boxd cluster sends one usage report an hour, with counts only, until a licence key with telemetry opt-out turns it off.
• Data in your account, no one to operate it: BYOC, from boxd or E2B.
• Many agents running LLM-written code, for users or for hours, in your own cloud or data centre: one VM per agent. Self-hosted boxd if you don't want to run Kubernetes; it's free up to 16 worker vCPUs, enough for a first cluster. Coder if you already run Kubernetes and want its IDE and governance features. E2B if you need Azure.

What to do next

Start with one GPU server and one machine: serve the model, add the egress rules, run one agent against it, and check in your network logs that nothing leaves that you haven't allowed. Then save the snapshot and scale out. Keep all three layers patched; rerunning the installer's apply rolls out a new boxd release. To run the machine layer in your own cloud or on your own servers, self-host boxd from the open installer.

Michiel Voortman

PostShare⎘Copy link