Chat

The Chat function of the Together Python Library allows you to easily chat with any of the models in Together API with a single command. With pre-structured prompts and history, the chat function allows for seamless back-and-forth interactions with your favorite models from the command line.

See all commands with:

together chat --help

To start an interactive chat session, simply run:

$ together chat

This will start a chat session with the default model togethercomputer/RedPajama-INCITE-7B-Chat.

You can specify a different model with --model or -m. See the List Models section for how to list all available models. Here's an example:

$ together chat -m togethercomputer/RedPajama-INCITE-Chat-3B-v1

The prompt format for chat looks like this:

"{HISTORY}{user_id}: {prompt}\n{bot_id}:"

The default value for is <human> for {user_id} and <bot> for {bot_id} which follows the prompt template for the laion/OIG dataset. These values can be assigned with --user_id and --bot_id and recommended to be changed to the model's format for best results.

together chat

  • max-tokens: Maximum number of tokens to generate per request. Defaults to 128.
  • stop: Tokens to stop generation at. Accepts multiple values. Defaults to [""] and adds user_id if a custom one is provided.
  • temperature: Determines the degree of randomness in the response. Default=0.7
  • top-p: Used to dynamically adjust the number of choices for each predicted token based on the cumulative probabilities. Default=0.7
  • top-k: Used to limit the number of choices for the next predicted word or token. Default=50
  • repetition-penalty: Controls the diversity of generated text by reducing the likelihood of repeated sequences. Higher values decrease repetition.