> ## Documentation Index
> Fetch the complete documentation index at: https://docs.together.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Request routing

> The dedicated container inference routing layer authenticates each request to your deployment and forwards it to a ready replica.

> [!NOTE]
>
> The routing layer covered on this page is in alpha and can change without notice. Its endpoint URL is not published yet. [Contact sales](https://www.together.ai/contact-sales) to get it, and share feedback by [contacting support](mailto:support@together.ai).

The [dedicated container inference](https://docs.together.ai/docs/together-deployments.md) routing layer authenticates each request to your deployment and forwards it to one ready replica. The routing layer carries HTTP, gRPC, and WebSocket traffic. Jobs you submit through the [queue API](https://docs.together.ai/docs/deployments-queue.md) don't pass through it.

## Endpoint URL

Every deployment has an alpha endpoint URL that ends in `/v1/deployment-request/<DEPLOYMENT_NAME>`. It has the same path as the URL that [`jig endpoint`](https://docs.together.ai/reference/cli/jig#jig-endpoint) prints but a different host. Only the alpha URL supports the features on this page.

To run the examples, set the `ENDPOINT_URL` environment variable to your alpha URL:

```bash theme={null}
export ENDPOINT_URL="<YOUR_ENDPOINT_URL>"
```

## What your container receives

The routing layer removes the `/v1/deployment-request/<DEPLOYMENT_NAME>` prefix and forwards the rest of the path to your container. For example:

| Request path | Path your container receives |
| - | - |
| `/v1/deployment-request/my-model/generate` | `/generate` |
| `/v1/deployment-request/my-model/v1/chat/completions?stream=true` | `/v1/chat/completions?stream=true` |
| `/v1/deployment-request/my-model` | `/` |

The request method, body, and query string arrive exactly as sent, and the path keeps its percent-encoding. A request body can be up to 32 MiB. gRPC requests have no size limit.

Responses stream back as your container writes them, so server-sent events (SSE) work without extra configuration.

## Authentication

Send a Together API key from the deployment's [project](https://docs.together.ai/docs/projects.md) as a bearer token in the `Authorization` header:

```bash theme={null}
curl "$ENDPOINT_URL/health" \
  -H "Authorization: Bearer $TOGETHER_API_KEY"
```

The routing layer verifies the key before it forwards the request, so your container doesn't need to validate one.

## How a replica is chosen

The routing layer picks a replica in two stages. First it picks a cluster at random from those that run your deployment, weighted by the number of ready replicas in each. A cluster with two ready replicas is picked twice as often as a cluster with one. Then it picks one of that cluster's ready replicas in round-robin order.

```mermaid theme={null}
flowchart LR
    app["Your application"] --> router["Together routing layer"]
    router -->|"2 ready replicas"| clusterA["Cluster A"]
    router -->|"1 ready replica"| clusterB["Cluster B"]
    clusterA --> a1["Ready replica"]
    clusterA --> a2["Ready replica"]
    clusterB --> b1["Ready replica"]
    clusterB -.-|"no requests"| b2["Replica not ready"]

    classDef client fill:#b65a7c,stroke:#76374d,stroke-width:1.5px,color:#ffffff;
    classDef neutral fill:#cbd5e1,stroke:#64748b,stroke-width:1.5px,color:#132133;
    classDef endpoint fill:#fc4c02,stroke:#b83702,stroke-width:1.5px,color:#ffffff;
    classDef warning fill:#c77700,stroke:#a05f00,stroke-width:1.5px,color:#ffffff;
    class app client;
    class router neutral;
    class clusterA,clusterB,a1,a2,b1 endpoint;
    class b2 warning;
```

Neither stage accounts for load. A replica that's busy with a long request gets new requests at the same rate as an idle one. If a replica can serve only a few requests at once, limit concurrency in your server, or send long-running work through the queue instead.

Two request headers change which replica a request reaches.

### Session affinity

To keep related requests on one replica, such as the turns of a conversation that reuse a cached prompt prefix, send the same `together-affinity-key` value with each request:

```bash theme={null}
curl "$ENDPOINT_URL/generate" \
  -H "Authorization: Bearer $TOGETHER_API_KEY" \
  -H "Content-Type: application/json" \
  -H "together-affinity-key: conversation-42" \
  -d '{"prompt": "Hello"}'
```

The key can contain only printable ASCII characters. To key on a value that can contain other characters, such as a user name, hash it first.

Affinity is best effort. A key keeps reaching the same replica until the deployment's set of ready replicas changes, for example because of a deploy, a scale-up, a scale-down, or a replica crash. When the set changes, only some keys move:

* Keys on a replica that stops being ready move to another ready replica, so their requests don't fail.
* A replica that becomes ready takes over a share of the keys from the others.
* The cluster pick is weighted by ready replicas, so a change in one cluster can move a key to a replica in another cluster.

### Replica pinning

Pinning sends a request to one specific replica. Use it to make routing decisions yourself, for example with your own load balancer in front of the deployment, or to debug a single replica. Set the `together-replica` header to the replica's ID:

```bash theme={null}
curl "$ENDPOINT_URL/health" \
  -H "Authorization: Bearer $TOGETHER_API_KEY" \
  -H "together-replica: <REPLICA_ID>"
```

To list a deployment's replica IDs, call the [get deployment](https://docs.together.ai/reference/deployments-get.md) endpoint. Each key of `replica_events` in the response is a replica ID.

A pinned request is never redirected. If the replica isn't ready or no longer exists, the request fails with a `410`. Replicas get new IDs when they're replaced, so fetch the IDs again after a deploy or a scale-down, and on any `410`.

When a request carries both headers, `together-replica` takes precedence over `together-affinity-key`.

## Shutdown

When a replica starts shutting down during a deploy or a scale-down, it stops getting new requests and receives `SIGTERM`. It then has a grace period, set by [`termination_grace_period_seconds`](https://docs.together.ai/reference/cli/jig#termination_grace_period_seconds), to finish the requests it's already serving before Together stops it. Handle `SIGTERM` by finishing in-flight requests before exiting, and set the grace period longer than your longest request.

The routing layer can also close long-lived connections, such as WebSockets and gRPC streams, during its own maintenance. Have your clients reconnect when a connection drops.

## gRPC

Together terminates TLS inside the cluster that runs your replica and forwards gRPC requests from there to your container over HTTP/2 (h2c). Your server must accept h2c connections on its [`port`](https://docs.together.ai/reference/cli/jig#port). It treats a request as gRPC when its `Content-Type` is `application/grpc`, optionally followed by a codec suffix such as `+proto` or by parameters after a `;`.

Connect your client to the endpoint URL's host on port 443 over a secure channel. Every gRPC method path must start with `/v1/deployment-request/<DEPLOYMENT_NAME>`. gRPC clients send `/<package.Service>/<Method>`, so add the prefix with a client interceptor. The routing layer removes the prefix before forwarding, so your server sees the method path it expects. A client that can't rewrite method paths can't reach the deployment.

This example calls the standard gRPC health service and needs the `grpcio` and `grpcio-health-checking` packages. Your own stubs use the same intercepted channel:

```python theme={null}
import collections
import os
from urllib.parse import urlparse

import grpc
from grpc_health.v1 import health_pb2, health_pb2_grpc

endpoint = urlparse(os.environ["ENDPOINT_URL"])
prefix = endpoint.path.rstrip("/")


class CallDetails(
    collections.namedtuple(
        "CallDetails",
        "method timeout metadata credentials wait_for_ready compression",
    ),
    grpc.ClientCallDetails,
):
    pass


class PrefixInterceptor(grpc.UnaryUnaryClientInterceptor):
    def intercept_unary_unary(self, continuation, details, request):
        new_details = CallDetails(
            prefix + details.method,
            details.timeout,
            details.metadata,
            details.credentials,
            details.wait_for_ready,
            details.compression,
        )
        return continuation(new_details, request)


credentials = grpc.composite_channel_credentials(
    grpc.ssl_channel_credentials(),
    grpc.access_token_call_credentials(os.environ["TOGETHER_API_KEY"]),
)
channel = grpc.intercept_channel(
    grpc.secure_channel(f"{endpoint.hostname}:443", credentials),
    PrefixInterceptor(),
)

stub = health_pb2_grpc.HealthStub(channel)
print(stub.Check(health_pb2.HealthCheckRequest()))
```

The interceptor covers unary calls. For streaming calls, apply the same rewrite in `UnaryStreamClientInterceptor`, `StreamUnaryClientInterceptor`, and `StreamStreamClientInterceptor`.

The default readiness check is an HTTP/1.1 request, which a gRPC-only server rejects. For such a server, implement the standard gRPC health service and set [`health_check_path`](https://docs.together.ai/reference/cli/jig#health_check_path) to `"/grpc.health.v1.Health"`. Together then checks readiness over gRPC.

When the routing layer rejects a gRPC request, it returns a gRPC status code on an HTTP `200` response, so your client raises a standard gRPC error. The [error table](#errors) lists the code for each case.

## WebSockets

The routing layer tunnels WebSocket connections that open with an HTTP/1.1 `Upgrade` request. It removes the path prefix as it does for any other request, so a connection to `$ENDPOINT_URL/ws` reaches your server at `/ws`. WebSockets over HTTP/2 aren't supported.

The routing layer reads the API key only from the `Authorization` header, so the upgrade request must include that header. Browser WebSocket APIs can't set it, so connect from a server-side client. This example uses the `websockets` library, version 13.0 or later:

```python theme={null}
import asyncio
import os

from websockets.asyncio.client import connect

url = os.environ["ENDPOINT_URL"].replace("https://", "wss://", 1) + "/ws"
headers = {"Authorization": f"Bearer {os.environ['TOGETHER_API_KEY']}"}


async def main():
    async with connect(url, additional_headers=headers) as websocket:
        await websocket.send("Hello")
        print(await websocket.recv())


asyncio.run(main())
```

## Errors

The routing layer returns these statuses when it rejects a request or can't deliver it. gRPC clients get the code in the gRPC status column instead. The routing layer's rejections have an empty body, while responses from your container pass through unchanged, including error responses. To tell the two apart, give your error responses a body.

| HTTP status | gRPC status | Cause | What to do |
| - | - | - | - |
| `400` | `2` (`UNKNOWN`) | The `together-affinity-key` value contains characters outside printable ASCII. | Send a printable ASCII key, or hash the value first. |
| `401` | `16` (`UNAUTHENTICATED`) | The request has no `Authorization: Bearer` header, or the API key isn't valid. | Send a valid Together API key. |
| `403` | `7` (`PERMISSION_DENIED`) | The API key belongs to a different project than the deployment. | Use a key from the deployment's project. |
| `404` | `12` (`UNIMPLEMENTED`) | The path doesn't start with `/v1/deployment-request/<DEPLOYMENT_NAME>`, or no deployment with that name is running. | Check the deployment name and, for gRPC, the method-path prefix. |
| `410` | `5` (`NOT_FOUND`) | `together-replica` names a replica that isn't ready or no longer exists, or its value contains characters outside printable ASCII. | Get current replica IDs from the [get deployment](https://docs.together.ai/reference/deployments-get.md) endpoint, or remove the header. |
| `413` | Doesn't apply | The request body is larger than the [size limit](#what-your-container-receives). | Send a smaller body. |
| `502` | `14` (`UNAVAILABLE`) | No response came back from a replica, because a connection failed or closed before the response headers arrived. | Check that your container listens on `port` and serves the request's protocol, and check `jig logs`. If your container is healthy, retry the request. If the error persists, [contact support](https://portal.usepylon.com/together-ai/forms/support-request). |
| `503` | `14` (`UNAVAILABLE`) | The deployment has no ready replica, or Together couldn't verify the API key. | Check `jig status` and your health check. If a replica is ready, retry the request. |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.