AI Evaluations UI

Introduction

This guide explains how to perform evaluations using the Together AI UI.

For a comprehensive guide with detailed parameter descriptions and API examples, see AI Evaluations.

Navigate to https://api.together.ai/evaluations and click “Create Evaluation”.

Upload your dataset or select one from your library. Preview your dataset content in the “Dataset Preview” section.

Type	Description
Classify	Categorizes input into one of the provided categories
Score	Evaluates input and produces a score within a specified range
Compare	Compares responses from two models to determine which performs better

Configure the judge model that will evaluate your inputs:

Field	Type	Required	Description
`judge model`	string	Yes	The model used for evaluation
`system template`	Jinja2 template	Yes	Instructions for the judge to assess the data

Classify parameters:

Field	Type	Description
`labels`	list of strings	Categories for classification. Mark each as ‘pass’ or ‘fail’ for statistics
`model_to_evaluate`	object or string	Model configuration or dataset column name

Score parameters:

Field	Type	Description
`min_score`	float	Minimum score the judge can assign
`max_score`	float	Maximum score the judge can assign
`pass_threshold`	float	Score at or above which is considered “passing” (optional)
`model_to_evaluate`	object or string	Model configuration or dataset column name

Compare parameters:

Field	Type	Description
`model_a`	object or string	First model configuration or dataset column name
`model_b`	object or string	Second model configuration or dataset column name

Choose how to provide responses for evaluation:

Use when generating new responses for evaluation:

Field	Type	Required	Description
`model_name`	string	Yes	One of our supported models
`model_source`	string	Yes	`"serverless"`, `"dedicated"`, or `"external"`
`system_template`	Jinja2 template	Yes	Generation instructions (see Templates)
`input_template`	Jinja2 template	Yes	Input format, e.g., `"{{prompt}}"`
`max_tokens`	integer	No	Maximum tokens for generation
`temperature`	float	No	Temperature setting for generation
`external_api_token`	string	When external	API bearer token for external providers
`external_base_url`	string	No	Custom base URL for external APIs

Use when evaluating pre-existing data from your dataset. Simply specify the column name containing the data to evaluate.

When using model_source = "external":

Enter a supported shortcut (e.g., openai/gpt-5). See Supported External Models.
Provide your external_api_token for the provider.
Optionally set external_base_url for custom OpenAI chat/completions-compatible endpoints.

For dedicated endpoints, set model_source = "dedicated" and paste your endpoint ID into the model field. See Dedicated Inference.

Wait for your evaluation job to complete. The UI will show the current status of your job.

Once complete, you can: