Skip to main content
The metrics endpoint is in beta. The host and path are subject to change, and access may need to be enabled for your organization. Confirm availability with your Together AI contact before you build against it.
Every project with an active deployment already has a Grafana dashboard that Together AI hosts and maintains. Run your own Grafana when you want to customize the dashboard, route alerts to your on-call tools, chart endpoints next to your application’s metrics, or see every project in your organization in one place. In this tutorial, you’ll run Prometheus and Grafana locally with Docker Compose, scrape the Prometheus-compatible metrics endpoint, import the same dashboard the hosted Grafana uses, and add an alert rule on the 5xx error ratio.
This tutorial uses Grafana, but any observability tool that ingests Prometheus metrics can scrape the same endpoint. For Datadog, use its OpenMetrics integration.
The Grafana dashboard for Together AI dedicated endpoints, showing request rate by status, 5xx error ratio, in-flight requests, and latency percentile panels, with Endpoint and Deployment filter variables at the top.

Requirements

  • A Together API key set as TOGETHER_API_KEY in your terminal. Prometheus and the curl commands authenticate with this key.
  • The Together CLI installed and logged in with tg login.
  • Docker with the Compose plugin.
  • A dedicated endpoint serving traffic. The next section creates one if you don’t have one already.

Step 1: Deploy an endpoint and send traffic

The dashboard only shows data for endpoints that are serving requests. If you already have a dedicated endpoint taking traffic, skip to Find your organization ID. Deploy a small model on its default hardware:
Check its status with tg endpoints get, passing the deployment ID from the deploy output. Once the deployment reaches DEPLOYMENT_STATE_READY, send it a steady trickle of requests so every panel has data. Leave this loop running while you set up Grafana:
Replace your_project_slug/grafana-demo with the endpoint string from the deploy output. Keep "stream": true in the request body, because the time-to-first-token panels only count streaming requests. See the quickstart for the full deploy walkthrough.

Step 2: Find your organization ID

The metrics endpoint is scoped to your organization, so the scrape path needs your organization ID. Print it with the CLI:
You should see output like this:
Copy the org_ value. Then confirm that the metrics endpoint responds, replacing org_abc123 with your organization ID:
The response is in the plain-text Prometheus exposition format, with # TYPE lines followed by series like edge_inference_requests_rate5m{endpoint_id="ep_...",status_code="200"}. An empty response means no dedicated endpoint in the organization has served traffic recently.

Step 3: Run Prometheus and Grafana

Create a directory with three files: a Compose file, a Prometheus scrape config, and a Grafana data source definition. docker-compose.yml runs the two services:
docker-compose.yml
prometheus.yml tells Prometheus to scrape the metrics endpoint once a minute. Replace your_together_api_key and your_org_id with your API key and the organization ID from Step 2:
prometheus.yml
prometheus.yml contains your API key in plain text. Keep the directory out of version control, or move the key into a file referenced by credentials_file instead of credentials.
grafana-datasource.yml pre-configures Grafana with Prometheus as its data source, so you can select it when you import the dashboard:
grafana-datasource.yml
The timeInterval value must match the scrape_interval in prometheus.yml, because Grafana uses it to size rate() windows. Without it, Grafana assumes the default 15-second scrape interval, so the worker line of the Request rate by layer panel stays empty. Start the stack:
Open the Prometheus targets page at http://localhost:9090/targets and confirm that the together-endpoint-metrics target reports UP. The target stays in an unknown state until the first scrape completes, usually within a minute.

Step 4: Import the dashboard

Together AI publishes the dashboard in the together-cookbook repository. It requires Grafana 10.2 or later and works with any Prometheus data source.
  1. Download the dashboard JSON:
  2. Open Grafana at http://localhost:3000 and sign in. The default credentials are admin / admin.
  3. Go to Dashboards, select New, then Import dashboard.
  4. Upload together-dedicated-endpoints-dashboard.json, select the Prometheus data source when prompted, and select Import.
The dashboard loads with the Endpoint and Deployment variables at the top set to All. Most panels fill in after the first scrape. See Read the dashboard for what each row shows.

Step 5: Alert on errors

Add an alert rule that fires when more than 5% of requests return a 5xx error for five minutes straight. Grafana alert rules can’t use dashboard variables like $endpoint, so the rule restates the Error ratio (5xx, edge) panel’s query without them.
  1. Add a contact point, because a new Grafana instance has none that can deliver alerts. Go to Alerting > Notification configuration, select New contact point, and set it up for the channel you want alerts sent to, such as Slack, PagerDuty, or a webhook.
  2. Go to Alerting > Alert rules and select New alert rule.
  3. Name the rule, select the Prometheus data source, switch the query editor to Code, and enter:
  4. Set the alert condition to Is above 0.05.
  5. Create a folder with New folder and an evaluation group with New evaluation group (keep the default 1m interval), then set Pending period to 5m.
  6. Under Configure notifications, select your contact point, then select Save.
The query sums across every endpoint in your organization, so the rule fires on the error ratio of all endpoints combined. To alert on a single endpoint, add endpoint_id="ep_abc123" to both selectors in the query.
Congrats! Your dedicated endpoint metrics now flow into your own Grafana, with an alert on errors.

Clean up

Stop the traffic loop with Ctrl+C, then shut down Prometheus and Grafana from the directory that holds docker-compose.yml:
If you deployed grafana-demo in Step 1, delete it to stop billing. Pass the endpoint ID from the deploy output:

Use Grafana Cloud

The same dashboard works in Grafana Cloud or any other Grafana instance you run. Point a Prometheus-compatible collector, such as Grafana Alloy or a Prometheus instance with remote_write, at the metrics endpoint with the same scheme, path, and bearer credentials as the prometheus.yml from Step 3, then import the dashboard JSON against the data source that collector writes to.

Troubleshooting

  • The Prometheus target shows a timeout or context deadline error: Scraping an organization with many endpoints can take longer than scrape_timeout allows. Raise scrape_interval and scrape_timeout together, for example to 120s and 115s, and set timeInterval to the new scrape_interval.
  • The scrape returns 401: The API key must belong to the organization in the scrape path. Run tg whoami --api-key "$TOGETHER_API_KEY" to see the key’s organization. Without --api-key, a logged-in CLI reports the organization of your tg login session instead.
  • Panels are empty: Confirm that the endpoint is serving requests, the Prometheus target is UP, and the dashboard time range covers the last few minutes. The worker line of Request rate by layer stays empty until two scrapes have completed.
  • Time-to-first-token (TTFT) panels are empty but others populate: The edge TTFT panel only counts streaming requests. Send requests with "stream": true.
  • The router TTFT panel is empty: This panel depends on a newer router build that’s still rolling out. Until the rollout completes, use the worker TTFT panel instead.

Next steps

Metrics and dashboards

Find the hosted Grafana dashboard and the full metric and label tables.

Configure autoscaling

Act on what the dashboard shows by tuning replica scaling.