10 open‑source projects, timed on four platforms.
GitHub-hosted runners, Blacksmith, Vercel CI and boxd, every machine 2 vCPU. The same commands on each, timed from the push to a green check.
GitHub and Blacksmith as one job, Vercel CI as one task. boxd sharded, as it always runs.
boxd vs12× faster8× faster10× faster
boxd vs11× faster8× faster7× faster
boxd vs7× faster20× faster39× faster
boxd vs7× faster6× faster6× faster
boxd vs6× faster5× fasterover 7× faster
boxd vs6× faster5× faster8× faster
boxd vs5× faster6× faster6× faster
boxd vs4.1× faster2.6× faster4.2× faster
boxd vs2.5× faster1.9× faster3.4× faster
Each project on its own scale. Times from the push to a green check, medians of the measured runs. ×N is the shard count. 15m+ never finished within Vercel CI's 15-minute task limit. Prettier's Vercel CI tasks for its memory-heavy suite had 4 CPUs.
The medians
Every number
Medians, from the push to a green check. ×N is the number of test shards.
| GitHub-hosted | Blacksmith | Vercel CI | boxd | |||||
|---|---|---|---|---|---|---|---|---|
| Project | Tests | One job | Sharded | One job | Sharded | One task | Sharded | Sharded |
| Django | 19,899 | 12m 54s | 3m 06s×8 | 8m 35s | 2m 08s×8 | 11m 06s | 2m 46s×8 | 65s×8 |
| Prettier | 102,413 | 9m 38s | 2m 36s×16 | 7m 47s | 2m 05s×16 | 6m 22s | 2m 09s×16 | 55s×16 |
| GitHub CLI | 8,445 | 2m 10s | 2m 02s×8 | 5m 53s | 1m 49s×8 | 11m 40s | 4m 21s×8 | 18s×8 |
| ripgrep | 1,222 | 73s | 2m 09s×8 | 56s | 65s×8 | 73s | 81s×8 | 11s×8 |
| Excalidraw | 2,456 | 9m 48s | 2m 39s×16 | 8m 20s | 2m 13s×16 | 8m 48s | 1m 54s×32 | 89s×32 |
| React | 6,990 | 13m 03s | 3m 33s×16 | 11m 05s | 2m 56s×16 | 15m+ | 4m 17s×32 | 2m 08s×32 |
| Vue | 3,765 | 2m 32s | 86s×8 | 2m 13s | 80s×8 | 3m 13s | 1m 43s×8 | 25s×8 |
| Pydantic | 13,509 | 2m 35s | 1m 58s×8 | 2m 47s | 86s×8 | 2m 49s | 73s×8 | 29s×8 |
| Zod | 8,320 | 4m 54s | 2m 06s×8 | 3m 10s | 2m 19s×8 | 4m 59s | 2m 26s×8 | 72s×8 |
| Svelte | 7,879 | 6m 24s | 3m 38s×8 | 5m 02s | 2m 30s×8 | 8m 53s | 4m 41s×8 | 2m 36s×8 |
15m+ is a run that never finished, because Vercel CI stops a task after 15 minutes. Prettier's Vercel CI tasks that run its patterns-dirs suite had 4 CPUs and 8 GB, because 4 GB ran out of memory.
How we measured it
Each project's Linux lint, format, typecheck and unit tests, with the same commands on every platform, timed from a push to main until the check turns green.
Full method and costsubuntu-latest, as one job or one job per shard, each restoring its dependency cache.
The same workflows with only the runner label changed, on Blacksmith's cache. Sticky disks were left out.
The same commands as Turborepo tasks after one shared install: one task, or a checks task plus one per shard. Each push changed a file every task hashes, so every task ran.
One machine per test shard, created from main's last green snapshot with dependencies and build caches on disk.
No code change, the steady state between merges. GitHub and boxd ran on 8 Oct 2026, Blacksmith and Vercel CI on 28 Oct.
Caveats
We pushed commits with no code change.
That's boxd's best case, because everything in the snapshot is still valid. On a real change boxd redoes part of its lint, typecheck and build work, so expect a smaller gap.
GitHub CLI's tests are mostly skipped.
Go skips tests whose code didn't change. boxd skipped almost all of them, GitHub 216 of 250, Blacksmith and Vercel CI none, which is why their single jobs are the slowest there.
Blacksmith's times vary a lot.
The same React job took 9 to 16 minutes across three runs, so we show medians.
Vercel CI caches whole tasks.
On a commit that changes no file it restores every task and is green in about 25 seconds. Any code change reruns every task in these setups, and that's what we timed.
Vercel CI stops a task after 15 minutes.
React's one task hit that limit every time, so it shows as 15m+. Prettier's memory-heavy test suite ran on 4 CPUs and 8 GB there, because 4 GB ran out.