Metrics and dashboards

Dedicated model inference records latency, throughput, and utilization metrics for every endpoint and deployment. These are the same series that drive autoscaling.

Analytics dashboard

The Together AI console shows per-endpoint charts for requests, tokens per second, input tokens, output tokens, latency, and time to first token, built on the metric series below. Open an endpoint and select the Analytics tab. Switch between Usage and Errors, view the endpoint Total or break it down By deployment, and adjust the time range. Use the dashboard to monitor an endpoint at a glance and to compare deployments during an A/B test.

The Analytics tab for an endpoint in the Together AI console, with charts for requests, tokens per second, input tokens, and output tokens, plus Usage and Errors toggles, a Total and By deployment breakdown, and a time-range selector.

The charts populate once the endpoint starts serving requests.

Grafana dashboard

Together AI automatically creates and hosts a Grafana instance for each project with an active deployment. The prebuilt dashboard charts the available metrics for every endpoint in the project. Unlike the Analytics tab, the dashboard breaks latency down by layer of the request path and shows queueing overhead and key-value (KV) cache utilization. To open the dashboard, select Open Grafana at the top of the Endpoints page to see every endpoint in the project, or at the top of an endpoint’s page to see only that endpoint. Grafana opens in a new tab, already signed in. The button appears once the instance is ready. Anyone who can view the project can open Grafana. Together AI deletes the instance 30 days after the project’s last deployment stops.

Read the dashboard

Use the Endpoint and Deployment variables at the top to focus on one endpoint, or on one deployment during an A/B test or traffic split. The rows follow the request path from the client inward, matching the metric groups of the metrics endpoint:

Golden signals (edge): Request rate stacked by status code, the 5xx error ratio, and in-flight requests, all measured at the front-door proxy. This row is the client’s view of the endpoint, so check it first during an incident.
Latency, client-observed (edge): Request duration and time to first token as p50, p90, and p99. Edge latencies are in milliseconds. The time-to-first-token panel counts only streaming requests.
Latency, server-side (router and worker): At the router, request duration and the routing and queue overhead before the worker. At the worker, generation duration, time to first token, and time per output token. Router and worker latencies are in seconds, not milliseconds. A gap between edge latency and worker latency points at network or queueing overhead rather than model speed.
Throughput and tokens: Request rate by layer (edge, router, and worker), token throughput by token type, and tokens per request. Compare the request rates across layers to find where requests drop off.
Engine and cache (worker): KV cache utilization per deployment and the prefix cache hit rate. Sustained high KV cache utilization is a signal to scale up or out. A low hit rate means little prefix reuse across requests.

Customize the dashboard

The hosted Grafana signs everyone in as a viewer, so you can’t edit the dashboard, save a copy, or create new dashboards. To customize the dashboard, import it into your own Grafana.

Events

Each endpoint has an audit feed of events, newest first. It merges endpoint-scoped events with the deployment-scoped events for every deployment under the endpoint, so scale-ups, traffic shifts, readiness changes, and pauses across every deployment all surface here. Use it to trace what happened during an autoscaling event or a traffic-split change. In the console, open an endpoint and select the Logs tab to browse the feed, with columns for time, type, source, level, and message.

The Logs tab for an endpoint in the Together AI console, an event feed with Time, Type, Source, Level, and Message columns showing the deployment's lifecycle events.

To read the feed from the terminal, list events with the CLI. Pass the endpoint ID or name:

Shell

tg endpoints events ep_abc123

Optional flags narrow the feed:

--types restricts to specific event-type strings, comma-separated (for example deployment.scaled,condition.set).
--min-level sets the minimum severity: debug, info, warn, or error.
--since and --until bound a time range.
--subject-id filters to a single subject, such as a rollout ID.
--deployment-ids scopes to specific deployments, comma-separated.
--limit and --after paginate (max 10000, default 50). When more events remain, the CLI prints the --after command for the next page.

Add --json for the raw event objects, including fields the table view omits. See the endpoints CLI reference for the full flag list.

Endpoint mutation events such as endpoint.updated record when a change happened, but they don’t include a field-level diff. Keep configuration history in your deployment system if you need to reconstruct exactly what changed.

Deployment events

To follow a single deployment instead of the whole endpoint, pass its ID to the --deployment-ids filter on the same events feed:

Shell

tg endpoints events ep_abc123 --deployment-ids dep_abc123

Filtering by deployment ID excludes endpoint-scoped events such as endpoint.updated, so the output covers only the listed deployments’ lifecycles.

Prometheus-compatible metrics endpoint

The metrics endpoint is in beta. The host and path below are subject to change, and access may need to be enabled for your organization. Confirm availability with your Together AI contact before you build against it.

Scrape real-time, per-organization performance metrics for your dedicated endpoints in standard Prometheus format. The endpoint works with any Prometheus-compatible scraper, including Prometheus, Grafana Agent, the Datadog OpenMetrics integration, and Vector.

Authentication and URL

Metrics are served per organization at:

GET https://o11y-de2-metrics.cloud.together.ai/organizations/{org_id}/metrics

Authenticate with your API key as a bearer token. The endpoint is org-scoped by design, so it returns only your organization’s data:

Shell

curl -H "Authorization: Bearer $TOGETHER_API_KEY" \
  "https://o11y-de2-metrics.cloud.together.ai/organizations/$ORG_ID/metrics"

Prometheus scrape config

Point a Prometheus-compatible scraper at the endpoint:

scrape_configs:
  - job_name: together-endpoint-metrics
    scheme: https
    authorization:
      credentials: <TOGETHER_API_KEY>
    metrics_path: /organizations/<ORG_ID>/metrics
    static_configs:
      - targets: ["o11y-de2-metrics.cloud.together.ai"]

To chart the scraped series, follow Run your own Grafana. It runs Prometheus and Grafana with Docker Compose and imports the example dashboard, which has the same panels as your project’s Grafana dashboard.

Available metrics

Metrics are grouped by the stage of the request path they measure: the edge (front-door proxy), the router, and the worker (model server). Most request, token, and latency metrics end in _rate5m. Together AI precomputes them as per-second rates over the last 5 minutes and exposes them as gauges, so chart them directly, without wrapping them in rate(). Each metric has one of these types:

Rate: A per-second rate, such as requests per second.
Rate histogram: A latency or size distribution, exposed as _rate5m_bucket, _rate5m_sum, and _rate5m_count series that each hold a per-second rate. Pass the _bucket series to histogram_quantile() for percentiles, and divide _sum by _count for the mean.
Counter: A cumulative count ending in _total. Wrap it in rate().
Gauge: A point-in-time value.

For example, this query returns the p99 request duration at the edge, in milliseconds, for one endpoint:

histogram_quantile(0.99, sum by (le) (edge_inference_request_duration_ms_rate5m_bucket{endpoint_id="<ENDPOINT_ID>"}))

Edge (front-door proxy)

Metric	Type	Unit	Notes
`edge_inference_requests_rate5m`	Rate	requests per second	Broken down by `status_code`.
`edge_inference_request_duration_ms_rate5m`	Rate histogram	milliseconds	End-to-end request duration at the edge.
`edge_inference_ttft_ms_rate5m`	Rate histogram	milliseconds	Time to first token. Most meaningful with `is_streaming="true"`.
`edge_inference_inflight_requests`	Gauge	requests	Concurrent in-flight requests at the edge.

Router

Metric	Type	Unit	Notes
`router_inference_requests_rate5m`	Rate	requests per second	Broken down by `status_code`.
`router_inference_request_duration_seconds_rate5m`	Rate histogram	seconds	Request duration at the router.
`router_inference_ttft_seconds_rate5m`	Rate histogram	seconds	Time to first token.
`router_pre_worker_duration_seconds_rate5m`	Rate histogram	seconds	Routing and queue overhead before the worker.
`router_inference_inflight_requests`	Gauge	requests	Concurrent in-flight requests at the router.
`router_token_rate5m`	Rate	tokens per second	Broken down by `token_type` and `requester_organization_id`.
`router_tokens_per_request_rate5m`	Rate histogram	tokens	Broken down by `token_type`.

Worker (model server)

Metric	Type	Unit	Notes
`worker_inference_request_total`	Counter	requests	Broken down by `status_code`.
`worker_ttft_seconds_rate5m`	Rate histogram	seconds	Time to first token.
`worker_generation_duration_seconds_rate5m`	Rate histogram	seconds	Total generation duration.
`worker_tpot_seconds_rate5m`	Rate histogram	seconds	Time per output token (inter-token latency).
`worker_token_rate5m`	Rate	tokens per second	Broken down by `token_type`.
`worker_tokens_per_request_rate5m`	Rate histogram	tokens	Broken down by `token_type`.
`worker_engine_kv_cache_utilization`	Gauge	ratio (0-1)	Fraction of the engine’s KV cache capacity in use.
`worker_engine_cache_hit_rate`	Gauge	ratio (0-1)	Hit rate of the engine’s KV cache. Higher values mean more prefix reuse across requests.

Deprecated metrics

The following counters and histograms are deprecated, and Together AI plans to remove them. Their values reset when a replica restarts, scales down, or goes idle, so rate() over them reports false spikes. Each has a _rate5m replacement:

Deprecated metric	Replacement
`edge_inference_requests_total`	`edge_inference_requests_rate5m`
`edge_inference_request_duration_ms`	`edge_inference_request_duration_ms_rate5m`
`edge_inference_ttft_ms`	`edge_inference_ttft_ms_rate5m`
`router_inference_requests_total`	`router_inference_requests_rate5m`
`router_inference_request_duration_seconds`	`router_inference_request_duration_seconds_rate5m`
`router_inference_ttft_seconds`	`router_inference_ttft_seconds_rate5m`
`router_pre_worker_duration_seconds`	`router_pre_worker_duration_seconds_rate5m`
`router_token_count`	`router_token_rate5m`
`router_tokens_per_request`	`router_tokens_per_request_rate5m`
`worker_ttft_seconds`	`worker_ttft_seconds_rate5m`
`worker_generation_duration_seconds`	`worker_generation_duration_seconds_rate5m`
`worker_tpot_seconds`	`worker_tpot_seconds_rate5m`
`worker_token_total`	`worker_token_rate5m`
`worker_tokens_per_request`	`worker_tokens_per_request_rate5m`

Each replacement carries the same labels as the metric it replaces, except that the router histograms gain replica_id and the edge metrics lose deployment_region. To migrate a query, switch to the replacement and drop the rate() around it. For histograms, also add _rate5m before the _bucket, _sum, or _count suffix:

# Deprecated
sum(rate(edge_inference_requests_total{status_code=~"5.."}[5m]))
histogram_quantile(0.99, sum by (le) (rate(worker_ttft_seconds_bucket[5m])))

# Replacement
sum(edge_inference_requests_rate5m{status_code=~"5.."})
histogram_quantile(0.99, sum by (le) (worker_ttft_seconds_rate5m_bucket))

If you imported the example dashboard before the _rate5m metrics were available, import the current version, which queries the replacements.

Labels

Series carry labels that identify the resource and slice the data. Not every label appears on every metric.

Label	Description
`owner_organization_id`	Organization that owns the endpoint.
`owner_project_id`	Project that owns the endpoint.
`endpoint_id` / `endpoint_name`	The endpoint.
`deployment_id` / `deployment_name`	The deployment.
`deployment_service_id`	Internal service identifier for the deployment.
`model` (edge) / `model_id` (worker)	The served model.
`deployment_region` (edge, except `_rate5m` metrics)	Region the deployment runs in.
`edge_region` (edge)	Region of the edge proxy that served the request.
`replica_id`	The specific replica.
`status_code`	HTTP status code of the request.
`is_streaming`	Whether the request was streamed (`"true"` or `"false"`).
`token_type`	Token category, such as input or output.
`path`	Request path.
`requester_organization_id`	Organization that made the request.

Next steps

Run your own Grafana

Scrape the metrics endpoint into your own Prometheus and Grafana, then alert on errors.

Configure autoscaling

Pick a metric to autoscale a deployment on.

Route traffic

See how the endpoint routes requests across deployments.

Gate rollouts with metrics

Pause a rollout automatically when a new deployment regresses on live metrics.

GET STARTED

SERVERLESS

INFERENCE APIS

DEDICATED MODEL INFERENCE

DEDICATED CONTAINER INFERENCE

POST-TRAINING

FINE-TUNING

GPU CLUSTERS

CODE EXECUTION

ADMINISTRATION

Analytics dashboard

Grafana dashboard

Read the dashboard

Customize the dashboard

Events

Deployment events

Prometheus-compatible metrics endpoint

Authentication and URL

Prometheus scrape config

Available metrics

Edge (front-door proxy)

Router

Worker (model server)

Deprecated metrics

Labels

Next steps

Run your own Grafana

Configure autoscaling

Route traffic

Gate rollouts with metrics