Visualize endpoint metrics in Grafana

​

Requirements

• A Together AI API key set as TOGETHER_API_KEY in your terminal.
• The Together CLI installed.
• Docker with the Compose plugin.
• A dedicated endpoint serving traffic. The next section creates one if you don’t have one already.

​

Step 1: Deploy an endpoint and send traffic

tg beta endpoints deploy google/gemma-4-E4B-it \
  --endpoint grafana-demo

while true; do
  curl -s https://api-inference.together.ai/v1/chat/completions \
    -H "Authorization: Bearer $TOGETHER_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "your_project_slug/grafana-demo",
      "messages": [{"role": "user", "content": "Write a haiku about dashboards."}],
      "stream": true
    }' > /dev/null
  sleep 5
done

​

Step 2: Find your organization ID

tg whoami

     Project: My Project (proj_abc123)
Organization: My Organization (org_abc123)

curl -H "Authorization: Bearer $TOGETHER_API_KEY" \
  "https://o11y-de2-metrics.cloud.together.ai/organizations/org_abc123/metrics"

​

Step 3: Run Prometheus and Grafana

services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:latest
    volumes:
      - ./grafana-datasource.yml:/etc/grafana/provisioning/datasources/together.yml
    ports:
      - "3000:3000"
    depends_on:
      - prometheus

global:
  scrape_interval: 60s
  scrape_timeout: 55s

scrape_configs:
  - job_name: together-endpoint-metrics
    scheme: https
    authorization:
      credentials: your_together_api_key
    metrics_path: /organizations/your_org_id/metrics
    static_configs:
      - targets: ["o11y-de2-metrics.cloud.together.ai"]

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      timeInterval: 60s

docker compose up -d

​

Step 4: Import the dashboard

1. Download the dashboard JSON:
   curl -O https://raw.githubusercontent.com/togethercomputer/together-cookbook/main/third_party_integrations/Grafana_Dedicated_Endpoints/together-dedicated-endpoints-dashboard.json
   
2. Open Grafana at http://localhost:3000 and sign in. The default credentials are admin / admin.
3. Go to Dashboards, select New, then Import.
4. Upload together-dedicated-endpoints-dashboard.json, select the Prometheus data source when prompted, and select Import.

​

Step 5: Read the dashboard

• Golden signals (edge): Request rate stacked by status code, the 5xx error ratio, and in-flight requests, all measured at the front-door proxy. This row is the client’s view of the endpoint and the first place to look during an incident.
• Latency, client-observed (edge): Request duration and time to first token as p50, p90, and p99, computed in Grafana from histogram buckets. Edge latencies are in milliseconds.
• Latency, server-side (router and worker): Request duration at the router, pre-worker routing and queue overhead, and generation duration, time to first token, and time per output token at the worker. Router and worker latencies are in seconds, not milliseconds. A gap between edge latency and worker latency points at network or queueing overhead rather than model speed.
• Throughput and tokens: Request rate compared across the edge, router, and worker layers, token throughput by token type, and tokens per request. The layer comparison shows where requests drop when the layers disagree.
• Engine and cache (worker): KV cache utilization per deployment and the prefix cache hit rate. Sustained high KV cache utilization is a signal to scale up or out, and a low hit rate means little prefix reuse across requests.

​

Use a hosted Grafana instance

​

Troubleshooting

• The Prometheus target shows a timeout or context deadline error: Organizations with many endpoints can take longer than the default timeout to scrape. Raise scrape_interval and scrape_timeout together, for example to 120s and 115s.
• The scrape returns 401: The API key must belong to the organization in the scrape path. Check both against tg whoami.
• Panels are empty: Confirm the endpoint is serving requests, the Prometheus target is UP, and the dashboard time range covers the last few minutes. Rate panels stay empty until two scrapes have completed.
• Time-to-first-token panels are empty but others populate: The edge TTFT panel only counts streaming requests. Send requests with "stream": true.
• The Router TTFT panel is empty: This series depends on a newer router build. The panel fills in as the rollout completes, and the worker TTFT panel covers the same question in the meantime.

​

Next steps