Skip to main content
Dedicated model inference (DMI) bills based on the hardware your deployments run on, regardless of model or request volume:
  • Billed by the minute: A deployment bills for as long as it runs, not per token or per request. The model you serve affects cost only through the hardware it needs (a larger model requires more or bigger GPUs), not through how many tokens or requests you push through it.
  • Per replica: Each running replica bills independently. A deployment running three replicas bills three times the single-replica rate.
  • Stops when scaled down: A replica stops billing as soon as it scales down. A deployment scaled to zero replicas, or stopped, costs nothing.
Because cost tracks running replicas, you keep your cost down by running only as many replicas as you need for your workload, and by stopping a deployment or setting its replica bounds to zero when you don’t need it. Endpoints run until you stop them; there is no automatic idle shutdown at launch. See Configure autoscaling for details.

​
Supported hardware

The following table lists the available hardware types. Where a single-GPU per-hour price is listed, multi-GPU configs cost proportionally more (a four-GPU config costs four times the single-GPU rate). For hardware without a listed price, contact sales for a quote. Hardware and GPU count are set by the config you select when you create a deployment.

​
How scaling affects cost

Billing is proportional to the number of running replicas across all deployments in your project. For a given deployment, you control how much it costs with its replica bounds, and by stopping it when you don’t need it:
  • minReplicas: This sets the floor for a deployment’s cost. These replicas will run and bill continuously, so set it to the lowest count that meets your latency target.
  • maxReplicas: This sets the ceiling for a deployment’s cost. The deployment never bills for more than this many replicas, so set it to a high enough count to handle your peak traffic.
  • Stop when idle: Stop a deployment or set both replica bounds to zero when you don’t need it. It bills nothing while stopped, and you restart it (requiring a cold start) by raising the replica bounds.
See Configure autoscaling for more details.

​
On-demand vs. reserved

Dedicated model inference offers two pricing options:
  • On-demand: Pay the per-minute rate for as long as your replicas run, with no commitment. Capacity scales up and down within your replica bounds. Best for variable traffic and prototyping.
  • Reserved: Commit to capacity for a set term at a lower effective rate, with guaranteed hardware availability. Best for steady, predictable production traffic. To set up reserved capacity, contact us.

​
DMI vs. serverless

Serverless models bill per token, while dedicated model inference bills per-minute for each running replica, regardless of how many tokens you push through. When comparing the two, consider how busy a replica would be for your workload:
  1. Work out your DMI cost from the per-minute rate: A single H100 replica at $5.49/hour costs about $132/day, or roughly $3,950 over a 30-day month, if running continuously.
  2. Estimate your serverless cost at the same volume: Monthly tokens multiplied by the model’s serverless per-token price.
DMI is usually cheaper when a replica would stay busy most of the day. The fixed per-minute cost is spread across high throughput, and you also get reserved capacity and predictable latency. Serverless is usually cheaper when traffic is low or bursty enough such that a dedicated replica would be sitting idle most of the time. You pay only for the tokens you use, so you lose nothing during an idle window. Stopping a deployment when it’s idle narrows the gap, but won’t help if your deployment receives steady low-volume traffic around the clock.

​
Next steps

Manage deployments

Create and manage deployments to serve your model.

Configure autoscaling

Control cost with replica bounds.