NVIDIA Reports 1.2%–4.3% Latency Overhead for Confidential B200 Inference
A controlled eight-GPU test retained 96.1%–98.2% of baseline output throughput after TensorRT-LLM adaptations, but the figures remain vendor-specific.
Edited by Tyronne Panaino
NVIDIA published a controlled confidential-computing comparison on September 22 for large-model inference on one eight-GPU DGX B200 system. The company reports that its confidential configuration retained 96.1%–98.2% of the non-confidential baseline's output-token throughput, while mean time per output token carried 1.2%–4.3% overhead across the tested concurrency range.
The result matters to infrastructure teams that want to protect sensitive prompts, model context and workloads while data is in use. It is also a tightly scoped vendor measurement, not a general guarantee for every model, cluster or serving pattern.
What NVIDIA measured
The test used eight NVIDIA B200 GPUs and the `nvidia/DeepSeek-R1-0528-NVFP4` model with the TensorRT-LLM PyTorch backend. NVIDIA disclosed a 32K-token input, a 1K-token output, tensor parallelism across eight GPUs, FP8 key-value cache and concurrency levels of 1, 2, 4, 8 and 16.
NVIDIA says it held the model, hardware, framework version, sequence lengths, parallelism and concurrency constant, changing only whether confidential computing was enabled. Under those conditions, confidential mode retained between 96.1% and 98.2% of baseline output-token throughput. Mean time per output token, a decode-latency measure, was between 1.2% and 4.3% above the baseline.
Those percentages describe relative performance within NVIDIA's disclosed setup. The source does not provide an independent reproduction, an absolute cost comparison or evidence that the same range will hold for different sequence lengths, batching policies, models, drivers or network topologies.
Why the runtime needed confidential-computing adaptations
NVIDIA explains that host-to-device transfers in the tested B200 confidential configuration pass through a software-encrypted bounce buffer because the GPU cannot directly access protected confidential-VM memory. TensorRT-LLM therefore selects pageable memory for affected paths and moves repeated token and sampling-data readback to an asynchronous worker so protected copies do not block the main decode scheduler.
The company also reports a timing change for kernel selection. In its tested configuration, CUDA-event timestamps were unstable for the autotuner, so TensorRT-LLM used the GPU's `%globaltimer` for tactic measurements under confidential computing while retaining CUDA events outside that mode.
A separate constraint affects multi-GPU communication: NVLink SHARP multicast was unavailable in the B200 confidential configuration. NVIDIA says frameworks should detect that capability and select communication algorithms for the actual message size, topology and workload rather than assume the multicast path is present.
The evidence boundary
The NVIDIA Technical Blog report is a detailed primary source for the configuration, methodology and figures NVIDIA chose to disclose. It is not independent validation of either the performance result or the wider security properties of the system.
The next useful checkpoint is reproduction on other workloads and software versions, with absolute throughput, latency distribution, resource use and operating cost reported alongside the relative percentages. Security attestation and threat-model evidence are separate questions that this performance comparison does not resolve.
Status
Confirmed. NVIDIA published the controlled comparison and the stated TensorRT-LLM adaptations. Internal confidence is medium because the measurements come from one first-party test and have not been independently corroborated here.
Sources
Update note: Last reviewed 2026-09-23. We will revise this post if independent reproductions or materially different NVIDIA results become available.
Sources
- NVIDIA Technical Blog — official
Drafted with AI assistance from source briefs; reviewed for citation completeness and label accuracy.