The routing layer covered on this page is in alpha and can change without notice. Its endpoint URL is not published yet. Contact sales to get it, and share feedback by contacting support.
Endpoint URL
Every deployment has an alpha endpoint URL that ends in/v1/deployment-request/<DEPLOYMENT_NAME>. It has the same path as the URL that jig endpoint prints but a different host. Only the alpha URL supports the features on this page.
To run the examples, set the ENDPOINT_URL environment variable to your alpha URL:
What your container receives
The routing layer removes the/v1/deployment-request/<DEPLOYMENT_NAME> prefix and forwards the rest of the path to your container. For example:
The request method, body, and query string arrive exactly as sent, and the path keeps its percent-encoding. A request body can be up to 32 MiB. gRPC requests have no size limit.
Responses stream back as your container writes them, so server-sent events (SSE) work without extra configuration.
Authentication
Send a Together API key from the deployment’s project as a bearer token in theAuthorization header:
How a replica is chosen
The routing layer picks a replica in two stages. First it picks a cluster at random from those that run your deployment, weighted by the number of ready replicas in each. A cluster with two ready replicas is picked twice as often as a cluster with one. Then it picks one of that cluster’s ready replicas in round-robin order. Neither stage accounts for load. A replica that’s busy with a long request gets new requests at the same rate as an idle one. If a replica can serve only a few requests at once, limit concurrency in your server, or send long-running work through the queue instead. Two request headers change which replica a request reaches.Session affinity
To keep related requests on one replica, such as the turns of a conversation that reuse a cached prompt prefix, send the sametogether-affinity-key value with each request:
- Keys on a replica that stops being ready move to another ready replica, so their requests don’t fail.
- A replica that becomes ready takes over a share of the keys from the others.
- The cluster pick is weighted by ready replicas, so a change in one cluster can move a key to a replica in another cluster.
Replica pinning
Pinning sends a request to one specific replica. Use it to make routing decisions yourself, for example with your own load balancer in front of the deployment, or to debug a single replica. Set thetogether-replica header to the replica’s ID:
replica_events in the response is a replica ID.
A pinned request is never redirected. If the replica isn’t ready or no longer exists, the request fails with a 410. Replicas get new IDs when they’re replaced, so fetch the IDs again after a deploy or a scale-down, and on any 410.
When a request carries both headers, together-replica takes precedence over together-affinity-key.
Shutdown
When a replica starts shutting down during a deploy or a scale-down, it stops getting new requests and receivesSIGTERM. It then has a grace period, set by termination_grace_period_seconds, to finish the requests it’s already serving before Together stops it. Handle SIGTERM by finishing in-flight requests before exiting, and set the grace period longer than your longest request.
The routing layer can also close long-lived connections, such as WebSockets and gRPC streams, during its own maintenance. Have your clients reconnect when a connection drops.
gRPC
Together terminates TLS inside the cluster that runs your replica and forwards gRPC requests from there to your container over HTTP/2 (h2c). Your server must accept h2c connections on itsport. It treats a request as gRPC when its Content-Type is application/grpc, optionally followed by a codec suffix such as +proto or by parameters after a ;.
Connect your client to the endpoint URL’s host on port 443 over a secure channel. Every gRPC method path must start with /v1/deployment-request/<DEPLOYMENT_NAME>. gRPC clients send /<package.Service>/<Method>, so add the prefix with a client interceptor. The routing layer removes the prefix before forwarding, so your server sees the method path it expects. A client that can’t rewrite method paths can’t reach the deployment.
This example calls the standard gRPC health service and needs the grpcio and grpcio-health-checking packages. Your own stubs use the same intercepted channel:
UnaryStreamClientInterceptor, StreamUnaryClientInterceptor, and StreamStreamClientInterceptor.
The default readiness check is an HTTP/1.1 request, which a gRPC-only server rejects. For such a server, implement the standard gRPC health service and set health_check_path to "/grpc.health.v1.Health". Together then checks readiness over gRPC.
When the routing layer rejects a gRPC request, it returns a gRPC status code on an HTTP 200 response, so your client raises a standard gRPC error. The error table lists the code for each case.
WebSockets
The routing layer tunnels WebSocket connections that open with an HTTP/1.1Upgrade request. It removes the path prefix as it does for any other request, so a connection to $ENDPOINT_URL/ws reaches your server at /ws. WebSockets over HTTP/2 aren’t supported.
The routing layer reads the API key only from the Authorization header, so the upgrade request must include that header. Browser WebSocket APIs can’t set it, so connect from a server-side client. This example uses the websockets library, version 13.0 or later: