Firecracker vs libkrun: Boot Time, Memory, and Sizing

The most common framing error is treating these two as competing implementations of one idea, then arguing about milliseconds. They are different integration models, and the model determines almost everything downstream.

The row that matters most is integration model. Firecracker hands you a process to supervise and a jailer to configure, which is exactly what you want when you are running a fleet: the failure domain is a process the supervisor can kill, restart, and account for. libkrun hands you a function call, which is exactly what you want when you are shipping a CLI or an agent runtime that has to sandbox code on someone else's laptop, and exactly what you do not want when a crash in the VM thread can take down the tool that was supervising it.

The Docker column is in the table to be dismissed on the correct grounds. It is not a faster point on the same curve. It is a different security tier: a shared host kernel means a kernel vulnerability or a runtime escape crosses the boundary, which is a defensible risk for a build cache and an indefensible one for code that arrived from an attacker-influenceable prompt.

Break a cold start into four stages and instrument each one. The published figures only cover stage two.

Two things fall out of that decomposition and they are the ones that change engineering decisions.

Stage 1 is usually the largest and it is not the hypervisor's fault. TAP device creation, IP allocation from a pool, and jailer chroot setup are host operations serialised on kernel locks that get contended when you spin up many VMs at once. A control plane that creates a TAP device on the request path will see stage 1 balloon at concurrency in a way that never appears in a single-VM benchmark. The fix is to pre-create network devices into a pool and hand them out, which is unglamorous and frequently cuts more milliseconds than any VMM tuning.

Stage 3 is where sloppy rootfs images hide. A full distribution rootfs running a real init system, spawning a logger, a DHCP client, and an SSH daemon will spend several hundred milliseconds doing work your sandbox does not need. A minimal image whose init is your entrypoint collapses stage 3 to near nothing. Most of the difference between a "microVMs are slow" experience and a "microVMs are fine" experience is here, not in the choice between Firecracker and libkrun.

If you take one operational action from this section: measure your own four stages before you compare runtimes, because there is a good chance the runtime is not your bottleneck.

Firecracker's documented overhead of under 5 MiB per microVM is a statement about the VMM process. It is genuinely impressive and it is genuinely not your per-VM cost. Here is the full accounting for one running microVM:

Three practical consequences.

First, a microVM configured with 512 MiB will eventually occupy something close to 512 MiB of host memory whether or not the workload needs it, because the guest kernel will happily fill unused memory with page cache. Memory ballooning exists to claw that back, and on a dense host it is not optional.

Second, rootfs sharing is a first-order capacity decision. Hundreds of VMs booting from a read-only copy-on-write base image share host page cache for the base. Hundreds of VMs each booting from their own full copy do not, and the host page cache becomes the binding constraint at a fraction of the density you planned for.

Third, the guest kernel is a config decision with a direct memory price. Stripping drivers you will never use in a virtio-only guest reduces both stage 2 boot time and resident kernel memory, and it is the single highest-leverage tuning knob available to someone who controls their own images.

Density planning fails when one profile is applied to everything. These four cover most of what we see in production.

The ephemeral exec row is the one that has changed most in the last two years, because that is the sandbox shape an AI coding agent needs: start from nothing, run one untrusted thing, exit, keep nothing. Cold start dominates and density is secondary, which is the profile where libkrun's embeddability and cross-platform support tend to beat Firecracker's density. The per-tenant row inverts every one of those weightings, which is why AWS built Lambda and Fargate on Firecracker and not on something embeddable.

If your workload is closer to "give me a hosted sandbox API and stop making me operate this", the managed options are compared in Modal vs E2B vs Daytona vs Vercel Sandbox.

This is the comparison people actually type into a search box, and it deserves a straight answer rather than a hedge.

The honest summary: Docker is faster, lighter, and easier, and none of those advantages address the threat model that makes people ask the question. If the code inside the boundary was written by a language model from a prompt a user controls, a shared kernel is the wrong isolation primitive and the cost of a separate kernel is the price of the correct one. If the code is your own, built in your own CI, running behind your own auth, a container is fine and a microVM is theatre.

There is a middle path worth naming: keeping the untrusted execution off a VM entirely by constraining what the code can reach, which is the approach discussed in sandboxing coding agents locally without a VM. It is weaker isolation and a much lower operational cost, and for a single-developer laptop that trade is often correct.

Snapshot and restore is the standard answer to cold start: capture a booted VM's memory and device state, then restore clones from it and skip kernel boot and init entirely. Restore lands well under a fresh boot because it skips the two most expensive stages.

Three inherited problems come with it, and all three have bitten real deployments.

Entropy is cloned. Every VM restored from one snapshot starts with an identical entropy pool. If the guest generates a key, a token, a session identifier, or a nonce shortly after restore, those values can collide across clones. This is a cryptographic failure, not a performance quirk, and it needs an explicit reseed on restore.

Time is frozen. The guest clock reads the snapshot moment until something corrects it. TLS certificate validation, token expiry checks, and anything log-ordered will misbehave until you resynchronise.

Network state is stale. Interfaces need reattaching, and anything the guest cached before the snapshot (DNS answers, connection pools, credentials with an expiry) is now wrong by however long the snapshot has been sitting.

Use snapshots for warm pools of short-lived, identical, non-secret-generating jobs. Do not use them for per-tenant VMs that hold credentials, and never restore a snapshot older than the shortest expiry inside it.

Vendor numbers are correct and unhelpful. Yours will differ because your rootfs, your kernel config, your network plumbing, and your concurrency are different. The harness that answers the question is small:

# Report p50 and p99 per stage, not a mean. Means hide the tail that pages you.
#
# Stage 1  t_orchestrate : API accepted        -> VMM process exists
# Stage 2  t_vmm         : VMM process exists  -> guest kernel init
# Stage 3  t_userspace   : guest kernel init   -> entrypoint executing
# Stage 4  t_ready       : entrypoint          -> first useful output

# Run each configuration three ways. The deltas between them are the finding.
#   cold  : drop host page cache first, nothing shared, nothing prewarmed
#   warm  : rootfs resident in host page cache, network devices pre-created
#   loaded: at your real target concurrency, not one VM at a time

for runtime in firecracker libkrun docker; do
  for mode in cold warm loaded; do
    for i in $(seq 1 200); do
      run_one "$runtime" "$mode"     # emit the four stage timestamps as one row
    done
  done
done

Two things this harness will tell you that a single-VM benchmark cannot.

The cold versus warm delta is frequently larger than the delta between runtimes. If dropping the host page cache costs you 300ms and switching hypervisors saves you 40ms, you now know where to spend the week.

The loaded row is where TAP device contention, IP pool exhaustion, and page-cache thrash show up. A runtime that looks 20% faster at concurrency 1 and falls apart at concurrency 200 is the wrong choice for a fleet and nobody finds that out from a published number.

Choose Firecracker when you run a Linux fleet, you want hundreds of concurrent VMs per host, and you can afford to operate rootfs images, kernel images, network plumbing, and a jailer. Its density and its operational track record at very large scale are not seriously contested, and the supervision model (a process you can kill and account for) is the right failure domain for a fleet.

Choose libkrun when the sandbox has to be embedded rather than supervised, or when it has to run on macOS. A developer tool that sandboxes code on the user's own Apple Silicon laptop cannot use Firecracker without shipping a Linux VM to hold it, and the resulting stack is worse than just using the library that already speaks Hypervisor.framework. You give up some density you probably were not going to use.

Choose Docker when the code is trusted. It is faster, lighter, and simpler, and the moment you find yourself writing a paragraph explaining why the shared kernel is acceptable for attacker-influenced code, you have chosen wrong.

At Particula Tech we size and operate this layer on-premise, usually for teams in regulated industries who need the isolation boundary documented as clearly as the throughput number. The capacity plans that survive contact with production are the ones built from a measured density test at the real guest size, with the network plumbing in the measurement. The ones that fail are built by dividing host RAM by guest RAM. The rest of our infrastructure and tooling work is collected on the AI development tools pillar.