Inference cost and serving economics

Connect the price of a GPU-hour to the cost of useful work. Compare explicit workload, utilization and contract assumptions.

The starting numbers are illustrative assumptions. Use measurements from your exact model, precision, serving engine, request lengths and latency target before making a decision. No throughput is inferred from GPU specifications.

Hosting and performance assumptions

GPUs in the measured systemGPUsRental rate per GPUUSD / GPU-hourOther hosting costs for the whole systemUSD / hourHours you plan to keep the systemhours in this scenarioAggregate output throughputoutput tokens / second, whole systemProductive serving time% of planned hours

Workload and successful tasks

Count every paid attempt, including retries. Success means an output meets your own named acceptance criterion. Set each deployment's measured acceptance rate separately using that same criterion.

Input tokens per attempttokensOutput tokens per attempttokensAttempts per day, including retriesattemptsWorking days in this workloaddaysCached share of input% of input tokensAttempts accepted as successful tasks% under your success criterionAPI attempts accepted as successful tasks% under the same success criterionTools and other variable costs per attemptUSD / attempt, both deployment options

API comparison prices

Enter the effective prices for this workload, accounting for its context tier and billing mode. These are your scenario inputs, not a live provider quote. Cache writes, searches, media, regional fees and taxes are omitted unless included in your other costs.

API input priceUSD / million tokensAPI output priceUSD / million tokensAPI cached-input priceUSD / million tokensA cached-input rate is recorded0 = use ordinary input rate; 1 = use cache rate

Interruptions and measurement basis

Use active throughput when utilization and restart losses are separate assumptions. A whole-period throughput measurement already includes both; the calculator applies neither penalty again.

Lost work during serving time% lost to restart or interruptionThroughput includes idle time and interruption loss0 = active; 1 = whole-period measurement

Latency requirements

Match the same percentile, concurrency and request lengths when comparing targets and measurements. Use 0 for an unknown measurement or an unused target. A missing or failed required latency measurement withholds hosted successful-task costs and contribution.

Time-to-first-token targetmilliseconds; 0 = no targetMeasured time to first tokenmilliseconds; 0 = unknownTime-per-output-token targetmilliseconds; 0 = no targetMeasured time per output tokenmilliseconds; 0 = unknown

Commitments and upfront cash

The minimum commitment is payable even if unused. Credit offsets GPU rent only; other hosting costs remain separately payable. Only the portion not credited against rent is treated as an additional fee. Unused credited balances remain cash exposure, not consumed expense. Refundable deposits and unspecified contract terms are excluded.

Minimum committed whole-system hourshours payable even if unusedUpfront paymentUSD, excluding refundable depositsUpfront payment credited against GPU rentUSD; excludes other hosting costs; the remainder is an additional fee

Selling prices and contribution margin

Enter your own selling prices to test break-even. Revenue includes all processed ordinary input, cached input and output under those prices. Contribution deducts modeled hosting and tool costs; it is not net profit or a forecast of sales.

Selling price: ordinary inputUSD / million tokensSelling price: outputUSD / million tokensSelling price: cached inputUSD / million tokens

Ownership comparison

This optional scenario spreads purchase price less assumed resale value over calendar life, plus the same other hosting costs. Supply power, cooling, CPU/RAM, support and maintenance there. Financing, taxes and downtime remain outside the estimate unless explicitly included. Performance is assumed equal; resale is not observed.

Whole-system purchase priceUSD; 0 hides ownership comparisonExpected whole-system resale valueUSD; an assumption, not a resale quoteOwnership lifecalendar hours

Record measurement context

Model revision, engine, precision, concurrency, latency percentile and success criterionThese notes stay in this page and in your downloaded record. They are omitted from copied URLs and interaction events.

Share URL

/tools/

What this scenario costs

Hosted scenario expense: $4,015.00Hosting plus all attempted-work tool costs; 730 payable system-hours.
API scenario expense: $105.00Assumed token rates plus the same per-attempt tool costs.
Hosted cost per successful task: $0.140928,500 assumed accepted tasks from 30,000 attempts.
API cost per successful task: $0.003728,500 API outputs meet the same assumed criterion.
Hosting cost per million output tokens at capacity: $3.0556At assumed productive throughput; includes this benchmark's input workload, excludes per-attempt tools.
Cost per productive GPU-hour: $11.00Paid idle time, unused commitment and additional upfront fees are included.
Contract cash exposure: $4,015.00Upfront payment plus remaining rental charges. Unused credited balance: $0.00.
Output capacity: 1,314,000,000Output tokens during the scenario; demand 15,000,000. This establishes no latency SLA.

How utilization changes hosting cost

Hosting cost under four productive-utilization assumptions, with the same system, commitment and interruption loss
Productive time	Hosting cost / million output tokens at capacity	Workload fits
25%	$6.1111	Yes, on throughput only
50%	$3.0556	Yes, on throughput only
75%	$2.037	Yes, on throughput only
100%	$1.5278	Yes, on throughput only

Measured reference workloads

These historical NVIDIA measurements are synthetic offline maximum-load references. They do not establish interactive latency, task quality, present-day software performance or cloud availability. Rental and API rates remain separate assumptions.

Custom scenario. No published measurement is being claimed for the current inputs.

Vendor-measured whole-system output throughput and exact request lengths
Workload	Input / output tokens	Output tokens / second
Short output	2,048 / 128	3,275.55
Longer output	2,048 / 2,048	9,462.43
Long input	20,000 / 2,000	1,340.69

All rows: nvidia/Llama-3.1-8B-Instruct-FP8, FP8, 1 GPU, DGX H100; H100 SXM 80GB; TensorRT-LLM, v0.21 documentation, PyTorch backend. Concurrency, per-row latency and measurement date were not published in this table. Source updated 2025-09-15; reviewed 2026-11-03. NVIDIA measurements and reproduction method. Source version 0c9430e; third-party terms apply.

How to read the calculation

Hosting expense equals payable whole-system hours times the GPU and other hourly rates, plus non-creditable upfront fees. Delivered output capacity equals the measured aggregate output rate times productive seconds. Successful-task cost divides all attempted-work expense by the outputs that meet your stated acceptance criterion.

Keep the model and revision, precision, serving-engine version, prompt and answer lengths, concurrency, hardware configuration, and latency target with your measurement. A throughput-only result cannot establish those requirements. This calculator does not test model quality or validate your contract.

Copied URLs contain the numeric inputs and a reference-workload ID where applicable. Changing measured performance or request lengths makes the scenario custom. Calculations run in your browser; the page uses the site's usual analytics and privacy policy.

Download the measured reference catalog · Reviewed research evidence

Background reading: Inference Economics on AI Wiki. AI Wiki is run by the same team as AI Compute Tracker.