The metrics endpoint is in beta. The host and path are subject to change, and access may need to be enabled for your organization. Confirm availability with your Together AI contact before you build against it.

Requirements
- A Together API key set as
TOGETHER_API_KEYin your terminal. Prometheus and thecurlcommands authenticate with this key. - The Together CLI installed and logged in with
tg login. - Docker with the Compose plugin.
- A dedicated endpoint serving traffic. The next section creates one if you don’t have one already.
Step 1: Deploy an endpoint and send traffic
The dashboard only shows data for endpoints that are serving requests. If you already have a dedicated endpoint taking traffic, skip to Find your organization ID. Deploy a small model on its default hardware:tg endpoints get, passing the deployment ID from the deploy output. Once the deployment reaches DEPLOYMENT_STATE_READY, send it a steady trickle of requests so every panel has data. Leave this loop running while you set up Grafana:
your_project_slug/grafana-demo with the endpoint string from the deploy output. Keep "stream": true in the request body, because the time-to-first-token panels only count streaming requests. See the quickstart for the full deploy walkthrough.
Step 2: Find your organization ID
The metrics endpoint is scoped to your organization, so the scrape path needs your organization ID. Print it with the CLI:org_ value. Then confirm that the metrics endpoint responds, replacing org_abc123 with your organization ID:
# TYPE lines followed by series like edge_inference_requests_rate5m{endpoint_id="ep_...",status_code="200"}. An empty response means no dedicated endpoint in the organization has served traffic recently.
Step 3: Run Prometheus and Grafana
Create a directory with three files: a Compose file, a Prometheus scrape config, and a Grafana data source definition.docker-compose.yml runs the two services:
docker-compose.yml
prometheus.yml tells Prometheus to scrape the metrics endpoint once a minute. Replace your_together_api_key and your_org_id with your API key and the organization ID from Step 2:
prometheus.yml
grafana-datasource.yml pre-configures Grafana with Prometheus as its data source, so you can select it when you import the dashboard:
grafana-datasource.yml
timeInterval value must match the scrape_interval in prometheus.yml, because Grafana uses it to size rate() windows. Without it, Grafana assumes the default 15-second scrape interval, so the worker line of the Request rate by layer panel stays empty.
Start the stack:
together-endpoint-metrics target reports UP. The target stays in an unknown state until the first scrape completes, usually within a minute.
Step 4: Import the dashboard
Together AI publishes the dashboard in the together-cookbook repository. It requires Grafana 10.2 or later and works with any Prometheus data source.-
Download the dashboard JSON:
-
Open Grafana at http://localhost:3000 and sign in. The default credentials are
admin/admin. - Go to Dashboards, select New, then Import dashboard.
-
Upload
together-dedicated-endpoints-dashboard.json, select the Prometheus data source when prompted, and select Import.
Step 5: Alert on errors
Add an alert rule that fires when more than 5% of requests return a 5xx error for five minutes straight. Grafana alert rules can’t use dashboard variables like$endpoint, so the rule restates the Error ratio (5xx, edge) panel’s query without them.
- Add a contact point, because a new Grafana instance has none that can deliver alerts. Go to Alerting > Notification configuration, select New contact point, and set it up for the channel you want alerts sent to, such as Slack, PagerDuty, or a webhook.
- Go to Alerting > Alert rules and select New alert rule.
-
Name the rule, select the Prometheus data source, switch the query editor to Code, and enter:
-
Set the alert condition to Is above
0.05. -
Create a folder with New folder and an evaluation group with New evaluation group (keep the default
1minterval), then set Pending period to5m. - Under Configure notifications, select your contact point, then select Save.
endpoint_id="ep_abc123" to both selectors in the query.
Congrats! Your dedicated endpoint metrics now flow into your own Grafana, with an alert on errors.
Clean up
Stop the traffic loop withCtrl+C, then shut down Prometheus and Grafana from the directory that holds docker-compose.yml:
grafana-demo in Step 1, delete it to stop billing. Pass the endpoint ID from the deploy output:
Use Grafana Cloud
The same dashboard works in Grafana Cloud or any other Grafana instance you run. Point a Prometheus-compatible collector, such as Grafana Alloy or a Prometheus instance withremote_write, at the metrics endpoint with the same scheme, path, and bearer credentials as the prometheus.yml from Step 3, then import the dashboard JSON against the data source that collector writes to.
Troubleshooting
- The Prometheus target shows a timeout or context deadline error: Scraping an organization with many endpoints can take longer than
scrape_timeoutallows. Raisescrape_intervalandscrape_timeouttogether, for example to120sand115s, and settimeIntervalto the newscrape_interval. - The scrape returns 401: The API key must belong to the organization in the scrape path. Run
tg whoami --api-key "$TOGETHER_API_KEY"to see the key’s organization. Without--api-key, a logged-in CLI reports the organization of yourtg loginsession instead. - Panels are empty: Confirm that the endpoint is serving requests, the Prometheus target is UP, and the dashboard time range covers the last few minutes. The worker line of Request rate by layer stays empty until two scrapes have completed.
- Time-to-first-token (TTFT) panels are empty but others populate: The edge TTFT panel only counts streaming requests. Send requests with
"stream": true. - The router TTFT panel is empty: This panel depends on a newer router build that’s still rolling out. Until the rollout completes, use the worker TTFT panel instead.
Next steps
Metrics and dashboards
Find the hosted Grafana dashboard and the full metric and label tables.
Configure autoscaling
Act on what the dashboard shows by tuning replica scaling.