Give a computer use agent a real desktop

Where to run a computer use agent so a wrong click or a hostile web page stays on a desktop that holds nothing you'd miss.

Hidde KehrerHidde Kehrer10 min readGuides
A machine reaches an arm out of its own casing to work a mouse and keyboard at a desk while a creature watches from a chair, arms folded.
On this page · 10 min

What a computer use agent does

A computer use agent is an AI agent that operates a computer the way a person does: it looks at the screen and acts with a mouse and keyboard. Your code sends the model a screenshot and a task, the model answers with one action such as "click at 640, 400", your code performs it on the desktop and sends a fresh screenshot, and the loop repeats until the model says it's done or you hit a step limit.

Both large model vendors ship this as a tool: Anthropic's computer use tool and OpenAI's computer use guide. The model chooses the actions but never touches a computer. You provide the desktop and its apps, so where it runs is your decision, separate from which model you use.

Here is a typical job. A support team still uses an old web CRM that has no API, and every refund means copying fields from a helpdesk ticket into four CRM screens. A computer use agent can do that copying if it has a desktop with a browser, both sites, and a login to each. If the CRM's pages were stable, a Playwright script would be simpler and more predictable; computer use pays off where there's no API and scripts keep breaking.

Why the agent needs a desktop of its own

The agent acts with whatever permissions the desktop has, and it treats everything on the screen as input. So a wrong click or a hostile page can reach anything the desktop can. If it runs on your laptop, that means your email, your files and your logged-in sessions. And because the model reads the screen, a web page can talk to it: Anthropic's docs warn that Claude sometimes follows instructions found in web pages or images even when they conflict with yours. A page with hidden text asking the agent to paste any passwords it knows into a form is the example Zapier's guide uses.

The vendors agree on the fix. Anthropic recommends a dedicated virtual machine or container with minimal privileges, no access to logins or sensitive data, internet access limited to an allowlist of domains, and a human who confirms anything with real-world consequences. OpenAI says the same: an isolated browser or VM, an allow list of sites, and screen content treated as untrusted.

That advice answers two separate questions: whose machine the agent runs on (a desktop that exists only for this task, so a mistake stays there), and what that desktop can reach. A clean desktop with your production keys on it and open internet access is still a way for data to leave.

There's also a design choice about what goes on the desktop. You can keep the loop and the model API key on your server and send only clicks and keystrokes to the desktop, which keeps the key away from anything a page could trick the agent into doing. Or you can put the whole agent on the desktop, which is simpler to start because everything sits in one place, but puts the key within the agent's reach.

Where to run a computer use agent

There are four places to run the desktop. They differ in the kernel, the core of the operating system (a container shares one with its host, a virtual machine or VM has its own), in how long the desktop lasts, and in how much you run yourself. What the desktop can reach is up to you in every option, and the checklist at the end applies to all four.

WhereGood forKernelHow long it lastsWhat you run
Your laptop, in a containerTrying it out, demosShared with its hostUntil you close the lidEverything
Provider-hosted computerPersonal tasksThe provider'sOne chat taskNothing
Hosted desktop sandboxAgents for users, long tasksIts own, on boxd and E2BFrom a capped session to as long as you keep itYour agent loop
Your own VMDesktops inside your networkIts ownAs long as you keep itThe VM, the desktop, the network rules

Your laptop, in a container. Anthropic's reference setup is a Docker container with a virtual display, a window manager, Firefox and LibreOffice. It's the fastest way to see the loop work. The container shares a kernel with its host (on a Mac, the Linux VM Docker runs), but the bigger risk is what you mount into it and log into from it. The agent also stops when your laptop sleeps.

The model provider's hosted computer. ChatGPT agent runs on a virtual computer that OpenAI hosts, with a visual browser, a text browser and a terminal, and you can take over for sensitive inputs such as logins. You run nothing. It works inside ChatGPT, not behind your own product.

A hosted desktop sandbox. You call an API, get a Linux desktop in the cloud, and send it actions, while the provider runs the machines and the live view. The strongest setup puts each desktop in its own microVM, a lightweight VM with its own kernel, so desktops don't share a kernel with each other. You can do this on boxd or E2B Desktop, and OpenAI's guide lists Daytona too, plus Browserbase and Browser Use for tasks that only need a browser. Providers differ most in how long a desktop may run. A running E2B sandbox on the Pro plan, for example, stops after 24 hours unless you pause and resume it.

Your own VM. A cloud VM with a desktop installed gives you full control, any operating system (a Windows VM if the app is a Windows client), and keeps the desktop inside your network. That cuts both ways, because an agent a web page can mislead now sits next to your internal systems. You also build the display, the remote view, the outbound limits and the cleanup yourself.

How to choose

Pick by who the agent works for and how long it runs:

  • You're trying computer use for the first time: run the reference container on your laptop and watch it. Don't log into anything real.
  • You want an assistant for your own tasks: the provider's hosted computer is the shortest path, and you can take over for logins.
  • You're building an agent that runs for your users, or for tasks that take hours: use a hosted desktop sandbox, one desktop per agent, and check how long a desktop may run.
  • The desktop must reach systems that only exist inside your network, or must run Windows: run your own VM there, and enforce the outbound limits from outside the VM, in your cloud's network rules, so nothing on the desktop can change them.

For the third case, ask each provider how long a desktop lasts, where its outbound limits are enforced, and whether real keys have to sit on it. On boxd, the machine keeps its files and processes for as long as the task needs, so a long task doesn't hit a session limit. Its egress allowlist (the list of hosts it may reach) is enforced outside the machine, so even root on the desktop can't widen it. API keys can be host-bound secrets: the machine only holds a bxds_ placeholder, which boxd swaps for the real value on HTTPS requests to the named host, so a misled agent can use that host but can't copy the key anywhere else. The egress control docs describe both. E2B is ahead on regions, a stated SOC 2 Type II report and an Apache-2.0 repository, so weigh those if your users or your compliance team need them.

Set up a desktop for your computer use agent

At the end you'll have a Linux desktop in its own VM that reaches only the helpdesk and the CRM, signed in by a person, driven from your terminal and watched live. The agent loop and model key stay on your machine.

1. Install the CLI and sign in.

Terminal
curl -fsSL https://boxd.sh/downloads/install.sh | sh
boxd auth login

The first line installs boxd on macOS or Linux, and the second signs you in.

2. Create an isolated machine for the agent.

Terminal
boxd machine new crm-agent --isolated

This creates a machine called crm-agent. The --isolated flag keeps it off your default network and away from your other machines' credentials and integrations, and it can't be undone later. The machine still reaches the internet until you set an allowlist.

3. Let it reach only the sites the task needs.

Terminal
boxd machine egress crm-agent crm.example.com,helpdesk.example.com

This replaces the machine's egress allowlist with those two hosts. Every other host is blocked and DNS lookups for other names are refused, so a page that tells the agent to send data elsewhere has nowhere to send it. Web apps often load scripts or a sign-in page from other hosts, so open each site once in step 4, see what fails to load, and add those hosts (wildcards such as *.example.com work).

4. Open the desktop and sign in as a person.

Terminal
boxd machine desktop crm-agent

This opens the machine's live desktop view in your browser. Press Ctrl+B (⌘B on a Mac), launch the Browser, and sign in to the helpdesk and the CRM yourself. Browser logins persist on the machine, so the agent never sees a password. Keep the view open to watch the agent and take over when it gets stuck.

5. Hold the screen at one size.

Terminal
boxd machine computer crm-agent screen-size 1280x800

Coordinates are screen pixels from the top left, so the screen must not change size while the model is clicking. 1280x800 is one of the sizes Anthropic recommends for web apps.

6. Take a screenshot and try a click.

Terminal
boxd machine computer crm-agent screenshot -o screen.png
boxd machine computer crm-agent click 640 400 --screenshot after.png

The first command saves the screen, with the CRM open, to screen.png. Put a field's coordinates in the second, and it clicks there and saves the result to after.png: one turn of the agent loop, by hand.

7. Connect the model. The boxd Python and TypeScript SDKs include ClaudeComputer and OpenAIComputer adapters that plug into each vendor's computer use tool: they scale screenshots for the model and map its coordinates back to the screen. Your loop runs on your server, so the model key never reaches the desktop. The computer use guide has the full loop for both vendors; give yours a step limit, as Anthropic's sample does.

8. Remove the machine when the task is done.

Terminal
boxd machine remove crm-agent -y

Until then the machine keeps its files, open windows and browser logins, so a task that runs for hours or waits for a person's approval picks up where it left off. Give the next task a fresh machine.

Before your agent runs unattended

Check these whichever option you picked:

  • The desktop exists only for this agent and holds no personal logins or files.
  • Its outbound traffic is limited to the sites the task needs, by rules the agent can't change.
  • Keys are either off the machine or bound to the one host that should receive them.
  • The loop has a step limit, and anything that spends money, sends messages or deletes data waits for a person.
  • Someone can watch the desktop and take over.

If your agents each need a desktop for hours at a time, run one machine per agent, and read sandboxes with a hardware boundary for why that boundary is a VM. To start, install the CLI:

Terminal
curl -fsSL https://boxd.sh/downloads/install.sh | sh

Frequently asked

What is a computer use agent?
An AI agent that operates a computer through its screen. Your code sends the model a screenshot and a task, the model answers with an action such as a click or typed text, your code performs it on a desktop, and the loop repeats until the task is done or a step limit is reached.
What is a desktop agent?
Another name for a computer use agent: an agent that works through a desktop's graphical interface with a mouse and keyboard instead of calling APIs. It's useful for software that has no API, such as an old CRM.
Can I run a computer use agent on my own laptop?
For a demo you watch, yes, inside a container such as Anthropic's reference setup. For unattended work, no: the agent acts with whatever access the desktop has, and web pages can carry instructions it may follow. Anthropic and OpenAI both recommend a dedicated VM or container with an allowlist of sites.
Is a Docker container enough to isolate a computer use agent?
It keeps the agent off your own desktop, but a container shares the host's kernel. For agents you run for other people, a VM per desktop, with its own kernel, is a stronger wall. Either way, limit which sites it can reach and keep keys off it.
What is the best computer use agent?
The model picks the actions, and Anthropic and OpenAI both offer a computer use tool. What you control is the desktop: run it somewhere that holds nothing you'd mind losing, with outbound traffic limited to the sites the task needs, and a person who confirms anything with real consequences.
Hidde KehrerHidde Kehrer
PostShare
Published
May 4, 2026
Reading time
10 min
Words
2,066
Last verified
Nov 12, 2026

Read next

Field notes

Subscribe for release notes and architecture write-ups

No spam, ever. Unsubscribe anytime.

Your inbox