Skip to main content
The routing layer covered on this page is in alpha and can change without notice. Its endpoint URL is not published yet. Contact sales to get it, and share feedback by contacting support.
The dedicated container inference routing layer authenticates each request to your deployment and forwards it to one ready replica. The routing layer carries HTTP, gRPC, and WebSocket traffic. Jobs you submit through the queue API don’t pass through it.

Endpoint URL

Every deployment has an alpha endpoint URL that ends in /v1/deployment-request/<DEPLOYMENT_NAME>. It has the same path as the URL that jig endpoint prints but a different host. Only the alpha URL supports the features on this page. To run the examples, set the ENDPOINT_URL environment variable to your alpha URL:

What your container receives

The routing layer removes the /v1/deployment-request/<DEPLOYMENT_NAME> prefix and forwards the rest of the path to your container. For example: The request method, body, and query string arrive exactly as sent, and the path keeps its percent-encoding. A request body can be up to 32 MiB. gRPC requests have no size limit. Responses stream back as your container writes them, so server-sent events (SSE) work without extra configuration.

Authentication

Send a Together API key from the deployment’s project as a bearer token in the Authorization header:
The routing layer verifies the key before it forwards the request, so your container doesn’t need to validate one.

How a replica is chosen

The routing layer picks a replica in two stages. First it picks a cluster at random from those that run your deployment, weighted by the number of ready replicas in each. A cluster with two ready replicas is picked twice as often as a cluster with one. Then it picks one of that cluster’s ready replicas in round-robin order. Neither stage accounts for load. A replica that’s busy with a long request gets new requests at the same rate as an idle one. If a replica can serve only a few requests at once, limit concurrency in your server, or send long-running work through the queue instead. Two request headers change which replica a request reaches.

Session affinity

To keep related requests on one replica, such as the turns of a conversation that reuse a cached prompt prefix, send the same together-affinity-key value with each request:
The key can contain only printable ASCII characters. To key on a value that can contain other characters, such as a user name, hash it first. Affinity is best effort. A key keeps reaching the same replica until the deployment’s set of ready replicas changes, for example because of a deploy, a scale-up, a scale-down, or a replica crash. When the set changes, only some keys move:
  • Keys on a replica that stops being ready move to another ready replica, so their requests don’t fail.
  • A replica that becomes ready takes over a share of the keys from the others.
  • The cluster pick is weighted by ready replicas, so a change in one cluster can move a key to a replica in another cluster.

Replica pinning

Pinning sends a request to one specific replica. Use it to make routing decisions yourself, for example with your own load balancer in front of the deployment, or to debug a single replica. Set the together-replica header to the replica’s ID:
To list a deployment’s replica IDs, call the get deployment endpoint. Each key of replica_events in the response is a replica ID. A pinned request is never redirected. If the replica isn’t ready or no longer exists, the request fails with a 410. Replicas get new IDs when they’re replaced, so fetch the IDs again after a deploy or a scale-down, and on any 410. When a request carries both headers, together-replica takes precedence over together-affinity-key.

Shutdown

When a replica starts shutting down during a deploy or a scale-down, it stops getting new requests and receives SIGTERM. It then has a grace period, set by termination_grace_period_seconds, to finish the requests it’s already serving before Together stops it. Handle SIGTERM by finishing in-flight requests before exiting, and set the grace period longer than your longest request. The routing layer can also close long-lived connections, such as WebSockets and gRPC streams, during its own maintenance. Have your clients reconnect when a connection drops.

gRPC

Together terminates TLS inside the cluster that runs your replica and forwards gRPC requests from there to your container over HTTP/2 (h2c). Your server must accept h2c connections on its port. It treats a request as gRPC when its Content-Type is application/grpc, optionally followed by a codec suffix such as +proto or by parameters after a ;. Connect your client to the endpoint URL’s host on port 443 over a secure channel. Every gRPC method path must start with /v1/deployment-request/<DEPLOYMENT_NAME>. gRPC clients send /<package.Service>/<Method>, so add the prefix with a client interceptor. The routing layer removes the prefix before forwarding, so your server sees the method path it expects. A client that can’t rewrite method paths can’t reach the deployment. This example calls the standard gRPC health service and needs the grpcio and grpcio-health-checking packages. Your own stubs use the same intercepted channel:
The interceptor covers unary calls. For streaming calls, apply the same rewrite in UnaryStreamClientInterceptor, StreamUnaryClientInterceptor, and StreamStreamClientInterceptor. The default readiness check is an HTTP/1.1 request, which a gRPC-only server rejects. For such a server, implement the standard gRPC health service and set health_check_path to "/grpc.health.v1.Health". Together then checks readiness over gRPC. When the routing layer rejects a gRPC request, it returns a gRPC status code on an HTTP 200 response, so your client raises a standard gRPC error. The error table lists the code for each case.

WebSockets

The routing layer tunnels WebSocket connections that open with an HTTP/1.1 Upgrade request. It removes the path prefix as it does for any other request, so a connection to $ENDPOINT_URL/ws reaches your server at /ws. WebSockets over HTTP/2 aren’t supported. The routing layer reads the API key only from the Authorization header, so the upgrade request must include that header. Browser WebSocket APIs can’t set it, so connect from a server-side client. This example uses the websockets library, version 13.0 or later:

Errors

The routing layer returns these statuses when it rejects a request or can’t deliver it. gRPC clients get the code in the gRPC status column instead. The routing layer’s rejections have an empty body, while responses from your container pass through unchanged, including error responses. To tell the two apart, give your error responses a body.