Benchmark · Task 1
Sandbox time to first command
From the moment you ask a provider for a brand new sandbox until echo ok comes back with exit code 0. Each provider gets 5 fresh sandboxes on its own default image.
01Results
Cold create to first command
ms · lower is better · p50 of 5 fresh sandboxes each
- 1Cloudflare Sandbox483 ms×1.00
- 2Fly.io Sprites623 ms×1.29
- 3Vercel Sandbox662 ms×1.37
- 4Freestyle976 ms×2.02
- 5Tensorlake1.92 s×3.97
- 6BoxLite Cloud4.07 s×8.42
- 7Upstash Box5.02 s×10.39
- 8Celesto6.94 s×14.37
- 1Cloudflare Sandbox467 ms×1.00
- 2Fly.io Sprites505 ms×1.08
- 3Vercel Sandbox632 ms×1.35
- 4Freestyle897 ms×1.92
- 5Tensorlake1.90 s×4.06
- 6Upstash Box2.76 s×5.90
- 7BoxLite Cloud3.75 s×8.03
- 8Celesto6.03 s×12.91
- 1Fly.io Sprites746 ms×1.00
- 2Cloudflare Sandbox888 ms×1.19
- 3Vercel Sandbox1.07 s×1.43
- 4Freestyle1.57 s×2.11
- 5Tensorlake2.49 s×3.34
- 6BoxLite Cloud4.43 s×5.94
- 7Upstash Box11.67 s×15.64
- 8Celesto17.30 s×23.19
Measured from one client over the public Internet. 4 more providers could not be measured yet; they are listed below with the reason, never as a zero.
02All metrics
Milliseconds, p50 with min to max underneath. Only successful replicates count toward the summary. Hover a row in the chart above for every sample.
| Provider | OK | cold_start | spawn | first exec after create | teardown |
|---|---|---|---|---|---|
| Cloudflare Sandbox cloudflare/debian-trixie (lite) | 5/5 | 483.0 467.1 to 888.4 | 310.7 297.7 to 716.3 | 172.3 169.4 to 202.5 | 235.0 198.8 to 261.3 |
| Fly.io Sprites sprites default image | 5/5 | 623.2 504.5 to 746.1 | 433.0 307.7 to 558.3 | 188.8 179.4 to 196.8 | 1043.7 628.6 to 3419.1 |
| Vercel Sandbox vercel/sandbox/universal (default) | 5/5 | 661.9 631.9 to 1067.1 | 371.5 358.6 to 764.0 | 279.2 269.9 to 327.2 | 333.0 305.7 to 467.0 |
| Freestyle freestyle default VM image | 5/5 | 975.6 896.6 to 1572.1 | 659.6 590.4 to 1272.0 | 304.0 296.8 to 316.1 | 379.3 362.9 to 455.7 |
| Tensorlake tensorlake default image (tensorlake/ubuntu-minimal) | 5/5 | 1918.8 1895.1 to 2490.0 | 851.4 842.3 to 1372.2 | 1063.0 1052.8 to 1117.8 | 221.6 218.9 to 226.0 |
| BoxLite Cloud boxlite cloud 'base' image | 5/5 | 4067.3 3752.2 to 4428.2 | 2336.8 2093.9 to 2683.7 | 1737.6 1462.4 to 1752.1 | 295.2 283.4 to 560.2 |
| Upstash Box upstash box default runtime (Debian 12 + Node) | 5/5 | 5019.3 2756.1 to 11667.4 | 4742.8 2488.1 to 11395.9 | 276.5 268.0 to 284.2 | 1685.0 499.9 to 2651.6 |
| Celesto celesto default template (scratch) | 5/5 | 6938.5 6030.2 to 17304.0 | 1572.7 1554.6 to 2464.1 | 5139.1 4475.6 to 14839.9 | 707.3 695.5 to 833.5 |
Provider notes
- Cloudflare Sandbox: Timed from this host through the bench Worker in ./cloudflare, so it includes one extra laptop-to-Worker hop.
- Fly.io Sprites: Fly.io Sprites via @fly/sprites. The token was minted from a Fly API token with SpritesClient.createToken().
- Vercel Sandbox: SDK create() is a single POST with no client-side polling. persistent: false, so no snapshot is kept.
- Freestyle: Created with an empty firewall (no network), matching Cloudflare's enableInternet: false.
- Tensorlake: REST API (the TS SDK breaks under Bun). POST /sandboxes waits until running by default, so spawn_ms covers boot. Free tier allows 1 concurrent sandbox.
- BoxLite Cloud: BoxLite Cloud on its 'base' image. Cloud create() returns with the box already running; start() is a no-op confirm.
- Upstash Box: SDK Box.create() waits for ready: if POST /v2/box returns 'creating' it polls GET /v2/box/:id every 2s, so spawn_ms may be quantized to ~2s steps.
- Celesto: Hosted Celesto Cloud computer via @celestoai/sdk. Computer.create() is one POST with no client-side polling.
03Not measured
A provider that could not be measured is shown here with the reason, not ranked as zero or last.
| Provider | Status | Why | Checked |
|---|---|---|---|
| E2B | blocked | New workspace is blocked until a $5.00 card verification payment is made. "This workspace is blocked until it is verified. Make a verification payment to keep using E2B." | 2026-11-11 |
| Runloop | blocked | Signup is invite only, and Runloop's docs say it is winding down its services (support for existing customers ends November 13, 2026). "Registration is currently by invitation only. Please email support@runloop.ai to join the waitlist." | 2026-11-11 |
| boat | blocked | API key works, but creating a sandbox needs a started plan (card checkout). "402 billing_required: Start the $20/month Boat plan to create sandboxes." | 2026-11-12 |
| Novita | blocked | Account has no balance yet. Novita's docs say the free credit unlocks after an onboarding survey. "SandboxError: 402: insufficient balance" | 2026-11-12 |
04Method
- Start a monotonic clock, then call the provider's create.
- Create a fresh sandbox on the vendor's default image. No custom templates, no warm pools we control.
- Run
echo okuntil it exits 0. Up to 40 tries, 250 ms apart, 2 s per try. - Run
uname -aonce as a second check. Not part of the headline number. - Destroy the sandbox. This always runs, even after a failure.
| Metric | Meaning |
|---|---|
| cold_start_ms | Headline. Clock start until echo ok exits 0. |
| spawn_ms | Clock start until the create call returns a handle. |
| time_to_first_exec_ms | Create returned until echo ok exits 0. |
| teardown_ms | Wall time of the destroy call. |
Only cold_start_ms is fair across providers. Some SDKs wait inside create until the sandbox is up, others return right away, so the split between spawn and first exec says more about the SDK than the platform.
Inspired by StarSling's sandbox benchmarks.
05Caveats
- Every number includes the network round trip from one client machine (darwin 27.0.0) to each provider. Your location will shift them.
- Default machine sizes differ by vendor. This ranks time to a usable sandbox, not machine performance.
- Cloudflare sandboxes can only be driven from inside a Worker, so that number goes through one small bench Worker and includes that extra hop.
- Upstash's SDK polls for readiness every 2 s inside create, so its numbers move in roughly 2 s steps.
- Five replicates is enough to rank wide gaps, not to separate providers that are within a few hundred ms.
- This measures create to first command from an SDK. It is not signup time, and it says nothing about disk, CPU or pipeline speed.
06Raw data
Every replicate, error and sandbox id behind this page:
- boxlite-cloud.json
- celesto.json
- cloudflare-sandbox.json
- fly-io-sprites.json
- freestyle.json
- tensorlake.json
- upstash-box.json
- vercel-sandbox.json
Harness: github.com/aiinfraengineer/ttfr-bench. Run it yourself with your own keys.