Confidential GPU Inference: The Real H100 Performance Tax

Start with the property, because half the confusion downstream comes from buying the wrong one. GPU confidential computing gives you three things: data in transit between the CPU's encrypted memory and the GPU is encrypted and integrity-protected; the host operator, hypervisor and other tenants cannot read GPU state through normal software paths, because the firewalls are up and the debug interfaces are down; and you get a cryptographic attestation that the silicon, firmware, driver and microcode you are talking to match a signed set of golden measurements.

It does not give you data residency, independence from the vendor's verification service, or encrypted HBM. NVIDIA's Hopper confidential-compute whitepaper is unambiguous on the last point: the on-package HBM memory is considered secure against common physical attack tools such as interposers, and is not encrypted. If your control narrative says "GPU memory is encrypted," a competent assessor will find the error. Write "encrypted transit across the interconnect, firewalled and physically-assumed-secure HBM, attested firmware and driver stack" instead. That is the discipline we apply when securing AI systems that handle sensitive data.

These are two layers, frequently conflated in vendor material. The CPU side is a confidential VM (Intel TDX or AMD SEV-SNP) that encrypts guest memory and shields the VM from the hypervisor. The GPU side is CC mode on the device, which raises hardware firewalls, disables performance counters and JTAG, and routes all DMA through encrypted bounce buffers in shared memory.

Neither is sufficient alone. A TDX VM with a plain passthrough GPU leaves weights and activations in plaintext on the device. A CC-mode GPU attached to a normal VM leaves the host free to read the prompt before it reaches the encrypted bus. The joined configuration is what the benchmarks below measure, and what NVIDIA's Blackwell attestation flow verifies by combining the GPU hardware report with CPU TEE measurements from SEV-SNP or TDX.

Two developer-facing consequences, both from NVIDIA's whitepaper. Pinned allocations from cudaHostAlloc() and cudaMallocHost() cannot be directly accessed by the GPU and are handled by unified memory with encrypted paging, so pinned memory access is slower. And cudaHostRegister() is not supported at all, returning an error code in CC mode. A serving layer that registers externally allocated host buffers fails outright rather than degrading.

Here is the whole spread, with the scope that makes each number honest.

The 2024 Hopper report is the one most often quoted as proof that confidential computing is free, and it deserves a careful read rather than a dismissal. Its own methodology section says TPS is measured by running the model with a batch size of 1. Its longest sequence bucket is 501 to 1500 tokens total. And its headline "nearly zero overhead" case is a 4-bit quantized Llama-3.1-70B producing 2.4822 tokens per second on an H100, where the GPU is so compute-bound that the encrypted transfer path is arithmetically irrelevant. Every one of those results is correctly measured, and none of them is a production serving regime. Batch 1 removes scheduler pressure, a 1500-token ceiling removes long-context prefill, and 2.5 tok/s removes the bus as a constraint by construction. Change any one and the number moves.

The most useful public dataset for a serving deployment is arXiv:2607.19353, published in May 2026. The setup is a single NVIDIA H100 80GB on a Google Cloud a3-highgpu-1g instance (26 vCPUs, 234 GB memory) running as an Intel TDX confidential instance, tested against Mistral-7B v0.1 and Qwen3-30B-A3B.

Fixed arrival rate results, at 10.0 req/s for Mistral and 5.0 req/s for Qwen:

Four things about that table need saying out loud, because each is a way to misread it.

The TTFT and latency rows carry no unit. The paper preserves whatever its source artifact reported and flags the mismatch as a known limitation: fixed-rate metrics labeled in seconds, closed-loop metrics in milliseconds, no normalization. Read literally, 279.56 would be seconds, implausible for a 7B on an H100 sustaining 10 req/s. Use the percentages. Anyone republishing those values with "ms" appended has added a unit the source does not contain.

The two gap columns use different arithmetic. Latency overhead is conventional, (CC minus Non-CC) divided by Non-CC. The throughput gap is (Non-CC minus CC) divided by CC, with the confidential value in the denominator. That is why the abstract's "17.7%" and the honest "you lose 15.0% of capacity" are both correct at once. The provisioning section turns that quirk into an advantage.

Global throughput here is not H100 capacity. 1458.65 tok/s is what that configuration produced at a fixed 10 req/s arrival rate, not at saturation, and Qwen's lower global figure follows from a 5 req/s arrival rate rather than a slower model.

Qwen3-30B-A3B is a mixture-of-experts model, 30B total with roughly 3B activated per token across 128 experts, 8 active. That is why the "larger" model posts higher per-request throughput (50.20 tok/s) than the 7B dense model (28.50 tok/s). The paper explicitly refuses the size-causes-overhead generalization, noting that sequence length, batching, runtime configuration, memory pressure and scheduler behavior all contribute.

The paper also states neither sequence lengths, precision, nor the serving engine, and reports no p90, p95 or p99, so it cannot tell you whether confidential mode widens the tail or only shifts the median. If your SLOs are written against p99, our guidance on setting TTFT and tokens-per-second targets applies to the measurement plan you will have to run yourself.

The steady-state percentages are the boring half. The closed-loop sweep is where the operational risk lives.

Mistral-7B, closed loop, all values p50 in milliseconds as labeled by the source:

Read the TTFT columns at c=32 and c=48. Confidential mode is faster, by 19.4% and 23.2%. That is in the paper and it is not an error. Below saturation, TTFT is dominated by queueing rather than by the encrypted transfer path, and the two configurations sit at different points on their own queueing curves. Past c=64 the sign flips and confidential mode runs 1.7% to 6.0% slower. A single TTFT delta measured at one load level therefore tells you almost nothing unless you also know whether you sampled pre-saturation, at the knee, or past it.

Now Qwen3-30B-A3B, on its own four-point grid:

Confidential mode flatlines at 0.84 req/s from c=32 onward. Non-confidential keeps climbing, 0.97 to 1.01, through c=40. The confidential ceiling is lower and it arrives earlier. That is the finding with real planning consequences: a deployment sized from low-load measurements looks fine in staging and under-provisions in production, because the two configurations diverge fastest exactly at the load where they stop scaling.

The c=32 row shows what crossing that knee costs. Confidential p50 TTFT is 20,823.1 ms against 2,874.1 ms non-confidential, roughly 7.2x. The paper is careful to say this single point does not represent steady-state overhead, because the two modes sit on opposite sides of their knees at that concurrency. Use it as an illustration of knee behavior, never as an overhead figure.

Scope it honestly: four concurrency points, one model, one GPU, one runtime. The direction is well supported and the magnitude is not. Measure your own.

Three independent primary sources agree on the mechanism, which is a stronger foundation than any single benchmark. NVIDIA's Hopper documentation states that the H100's DMA engines in a confidential environment may only access shared memory pages, so payloads stage through encrypted, signed bounce buffers before being decrypted inside the enclave. It puts a hard number on the resulting ceiling: CPU-GPU interconnect bandwidth is limited by CPU encryption performance to approximately 4 GB/s. It also states the rule plainly, that CC workloads perform close to non-CC mode when compute is large compared to input data, and that the interconnect limits throughput when it is not.

The 2024 Hopper report reaches the same conclusion from measurement. TTFT overhead consistently exceeds inter-token latency overhead (19.03% versus 7.67% on H100 with Llama-3.1-8B), which it reads as an I/O bottleneck rather than computation inside the TEE, and its throughput overhead falls monotonically as sequences lengthen: 7.16% short, 6.90% medium, 6.65% long.

NVIDIA's July 2026 Blackwell benchmark reproduces the gradient on newer silicon:

Longer inputs roughly halve the penalty. NVIDIA attributes Blackwell CC overhead to secure work submission latency (smaller units of work are more affected, and increasing work per GPU launch reduces the impact) plus reduced host-to-device bandwidth. Same mechanism, different generation.

Three caveats on that table, applied with the scrutiny any vendor benchmark deserves. It is HGX B300 Blackwell Ultra with Qwen 3.5-397B-A17B at FP8, running SGLang with CC-specific patches written to mitigate this exact overhead, so setting it against a stock H100 under TDX compares two hardware generations, two model scales, and a tuned stack against an untuned one. The claim "typically under 8%" is slightly exceeded by their own worst cell at -8.1%, and "up to 98%" describes a best case rather than a typical one. And while the surrounding prose says the table summarizes throughput, TTFT and TPOT overhead, and lists median TTFT under metrics collected, the published table carries throughput and TPOT columns only. TTFT is the metric that degrades most in both academic datasets. Note that when you cite the 98%.

Attestation is where confidential inference stops being a benchmark question and becomes an availability question.

What gets verified: the relying party challenges the GPU with a nonce of at least 128 bits and receives an evidence bundle of hardware, firmware, driver and microcode measurements, signed with a device attestation key. Device identity rests on a device-unique ECC-384 key pair burned into fuses at manufacture, with every private copy destroyed during manufacturing. The verifier checks the certificate chain against NVIDIA's CA, checks revocation via CRL or OCSP, compares the evidence against a signed reference integrity manifest of golden measurements, applies a policy, and issues an Entity Attestation Token as a JWT validated against an OpenID key set. Nonces are cached with the device certificate hash for a 24-hour TTL to block replay.

The cost is not latency. NVIDIA states that the handshake is typically a one-time startup event and that once the workload is running, attestation does not add latency to individual inference requests. No first-party millisecond figure for the handshake is published anywhere, so do not let one into a design document.

The cost is dependency. Remote verification is an outbound HTTPS call that fetches a reference manifest for your exact driver version, and NVIDIA documents the failure explicitly: any exception from the manifest service, a bundle not found for instance, results in GPU attestation failure and an unsupported driver version response. A routine driver upgrade can take your whole fleet out of attestation. Air-gapped deployments fall back to local verification, with NVIDIA's own caveat that stale local data regarding revocation status or verifier integrity may occur.

The design rule follows: attestation gates the boot, never the request.

# Boot-time admission gate. Call names are placeholders for your
# attestation client; the ordering is the part that matters.
import os


def admit_node() -> None:
    nonce = os.urandom(16)                    # spec floor is 128 bits
    evidence = collect_gpu_evidence(nonce)    # device cert + measurements
    token = verify(evidence)                  # remote service or local verifier

    if token is None or token.expired() or not token.claims_ok():
        raise AttestationFailed("node not admitted to the serving pool")

    unseal_model_weights(token)   # decryption keys released only after verify
    mark_ready()                  # only now does the load balancer see this node

Four operational rules follow from that shape, and each has bitten deployments we have reviewed:

The 2026 paper reports throughput gap as (Non-CC minus CC) divided by CC. As a statement of "how much capacity did we lose," that overstates the answer. As a statement of "how many more confidential GPUs do I need to match my current throughput," it is exactly right, because Non-CC divided by CC is precisely that multiplier. A 17.7% gap means 17.7% more GPUs. The capacity lost at a fixed GPU count is the other number, 15.0%.

import math

def cc_sizing(gpus_today: int, non_cc_tps: float, cc_tps: float) -> dict:
    gap = (non_cc_tps - cc_tps) / cc_tps        # the paper's reported "gap"
    return {
        "reported_gap_pct": round(gap * 100, 1),          # size on this
        "capacity_loss_pct": round((1 - cc_tps / non_cc_tps) * 100, 1),  # report this
        "gpus_required": math.ceil(gpus_today * (1 + gap)),
    }

# Mistral-7B v0.1, fixed 10 req/s, global token throughput
cc_sizing(8, 1458.65, 1239.26)
# {'reported_gap_pct': 17.7, 'capacity_loss_pct': 15.0, 'gpus_required': 10}

# Qwen3-30B-A3B, fixed 5 req/s, global token throughput
cc_sizing(8, 993.80, 820.31)
# {'reported_gap_pct': 21.1, 'capacity_loss_pct': 17.5, 'gpus_required': 10}

Applied across the published datasets, for a fleet of 8 GPUs held at constant throughput:

Use 15% to 25% as the initial reserve on Hopper, which is also the paper's own stated heuristic, then replace it with a measurement. Three things belong in that measurement plan, none optional: your real prompt and output length distribution (overhead falls as sequences lengthen), a concurrency sweep dense enough to locate the knee in both modes rather than one load point, and p95 or p99 rather than p50, because no public source tells you whether confidential mode widens the tail.

One structural note before extrapolating: every number above is single-GPU. On Hopper, multi-GPU confidential operation over NVLink uses encrypted bounce buffers in unprotected GPU memory that reduce throughput, and no figure for that reduction is published. Confidential inference with NVLink encryption across up to 8 GPUs is a Blackwell HGX B200 and B300 capability. If your model needs tensor parallelism across a Hopper node, you are in unmeasured territory and should treat any vendor estimate as a hypothesis.

We recommend confidential GPU inference in one specific situation: you must use rented GPUs and your threat model names the host operator. If that describes you, the 15% to 25% tax is a fair price for a property you cannot otherwise obtain. Four situations where it is the wrong tool.

Your requirement is residency rather than blindness. If the obligation is that data must not leave a jurisdiction or an estate, confidential computing does not deliver it. The bytes still sit on hardware you do not control, and remote attestation adds an outbound dependency on a vendor service. Regulators reading a data-flow diagram tend to notice both.

Your workload swaps models. Platforms that page models in and out of GPU memory are the regime where a published benchmark measured 45% to 70% higher throughput without confidential mode, plus roughly 50% higher GPU utilization. That is a different order of magnitude from the steady-state numbers, and it comes from weights repeatedly crossing a bus limited to approximately 4 GB/s. Consolidate to fewer resident models before paying it.

The tax pushes you past the ownership break-even. Add 15% to 25% more rented GPUs to a workload that was already marginal and the economics of owning hardware change materially. Work the numbers with the on-premise versus cloud AI security and cost comparison and the self-hosting break-even math before renewing a commitment. Across the regulated deployments we have reviewed, a private rack removes the threat model that confidential computing was purchased to address, without a throughput penalty, an attestation service dependency, or a driver upgrade that can fail closed.

You depend on GPU profiling in production. Performance counters and JTAG are off in full CC-On mode by design, so an operational practice that relies on kernel-level profiling in the deployed configuration ends the day you enable CC mode. Decide that consciously rather than mid-incident.

The rest of our work on threat models, deployment boundaries and evidence for regulated AI sits in the AI security pillar.

The one-line version: confidential GPU inference on H100 costs roughly a fifth of your throughput and a quarter of your time to first token, the tax lands on the boundary crossing rather than the arithmetic, and the number that will actually hurt you is the saturation knee moving left where your staging load test cannot see it.