Examples

Use these examples to learn best practices for using the inference API.

Suggest Edits

Table of Contents


Sentiment Analysis

You can use togethercomputer/RedPajama-INCITE-7B-Base for classification tasks with longer context. In this guide, you will use the model to classify the sentiment of any sentence as either positive, negative, mixed, or neutral.

  1. Begin the prompt with an instruction describing the task.
Label the sentences as either "positive", "negative", "mixed", or "neutral":
  1. After the instruction, provide a few examples with known labels.
Sentence: I can say that there isn't anything I would change.
Label: positive

Sentence: I'm not sure about this.
Label: neutral

Sentence: I liked some parts but I didn't like other parts.
Label: mixed

Sentence: I think the background image could have been better.
Label: negative
  1. Finally, append the tweet that the model should classify and prompt the model to complete the label.
Sentence: I really like it.
Label:
  1. Send the prompt to the API with any appropriate parameters. The code below shows an example in Python using the requests package.
import requests

endpoint = 'https://api.together.xyz/inference'
TOGETHER_API_KEY = os.getenv('TOGETHER_API_KEY')

res = requests.post(endpoint, json={
    "model": 'togethercomputer/RedPajama-INCITE-7B-Base',
    "prompt": """\
      Label the sentences as either "positive", "negative", "mixed", or "neutral":

      Sentence: I can say that there isn't anything I would change.
      Label: positive

      Sentence: I'm not sure about this.
      Label: neutral

      Sentence: I liked some parts but I didn't like other parts.
      Label: mixed

      Sentence: I think the background image could have been better.
      Label: negative

      Sentence: I really like it.
      Label:""",
    "top_p": 1,
    "top_k": 40,
    "temperature": 0.8,
    "max_tokens": 1,
    "repetition_penalty": 1,
}, headers={
    "Authorization": f"Bearer {TOGETHER_API_KEY}",
    "User-Agent": "<YOUR_APP_NAME>"
})
print(res.json()['output']['choices'][0]['text']) # ' positive'

Topic classification

RedPajama-INCITE-Instruct-7B can be used to classify the topic of text passages. In this guide, you will use the model to classify the topic of a news article as either World, Sports, Business, or Sci/Tech.

  1. Begin the prompt with an instruction describing the task.
Given a news article, classify its topic.
Possible labels: 1. World 2. Sports 3. Business 4. Sci/Tech
  1. After the instruction, provide a few examples with known labels.
Article: A nearby star thought to harbor comets and asteroids now appears to be home to planets, too.
Label: Sci/Tech

Article: Soaring crude prices plus worries about the economy and the outlook for earnings are expected to hang over the stock market next week during the depth of the summer doldrums.
Label: Business
  1. Finally, append the tweet that the model should classify and prompt the model to complete the label.
Article: Murtagh a stickler for success Northeastern field hockey coach Cheryl Murtagh doesn't want the glare of the spotlight that shines on her to detract from a team that has been the America East champion for the past three years and has been to the NCAA tournament 13 times.
Label:
  1. Send the prompt to the API with any appropriate parameters. The code below shows an example in Python using the requests package.
import requests

endpoint = 'https://api.together.xyz/inference'

res = requests.post(endpoint, json={
    "model": "togethercomputer/RedPajama-INCITE-Instruct-7B",
    "prompt": """
        Given a news article, classify its topic.
        Possible labels: 1. World 2. Sports 3. Business 4. Sci/Tech

        Article: A nearby star thought to harbor comets and asteroids now appears to be home to planets, too.
        Label: Sci/Tech

        Article: Soaring crude prices plus worries about the economy and the outlook for earnings are expected to hang over the stock market next week during the depth of the summer doldrums.
        Label: Business

        Article: Murtagh a stickler for success Northeastern field hockey coach Cheryl Murtagh doesn't want the glare of the spotlight that shines on her to detract from a team that has been the America East champion for the past three years and has been to the NCAA tournament 13 times.
        Label: """,
    "top_p": 1,
    "top_k": 40,
    "temperature": 0.8,
    "max_tokens": 1,
    "repetition_penalty": 1,
}, headers={
    "Authorization": "Bearer <YOUR_API_KEY>",
    "User-Agent": "<YOUR_APP_NAME>"
})
print(res.json())

Question answering

RedPajama-INCITE-Instruct-7B can be used to answer questions with reasoning on a user-provided context.

  1. Begin the prompt by providing the context.
Daniel travelled to the kitchen.
Mary grabbed the milk.
Daniel went back to the hallway.
John moved to the kitchen.
Daniel travelled to the office.
John grabbed the apple.
John journeyed to the garden.
John dropped the apple.
John moved to the hallway.
Mary dropped the milk.
Daniel went back to the bathroom.
Mary grabbed the milk there.
Mary went back to the garden.
Mary went to the hallway.
Mary left the milk.
John journeyed to the bedroom.
  1. Then, add a question followed by "Question:" and prompt the model to answer the question.
Question: Where was the milk before the hallway?
Answer:
  1. Sample response:
Answer: garden
  1. Send the prompt to the API with any appropriate parameters. The code below shows an example in Python using the requests package.
import requests

endpoint = 'https://api.together.xyz/inference'

res = requests.post(endpoint, json={
    "model": "togethercomputer/RedPajama-INCITE-7B-Instruct",
    "prompt": """\
        Daniel travelled to the kitchen.
        Mary grabbed the milk.
        Daniel went back to the hallway.
        John moved to the kitchen.
        Daniel travelled to the office.
        John grabbed the apple.
        John journeyed to the garden.
        John dropped the apple.
        John moved to the hallway.
        Mary dropped the milk.
        Daniel went back to the bathroom.
        Mary grabbed the milk there.
        Mary went back to the garden.
        Mary went to the hallway.
        Mary left the milk.
        John journeyed to the bedroom.
        Question: Where was the milk before the hallway?
				Answer: """,
    "top_p": 1,
    "top_k": 40,
    "temperature": 0.1,
    "max_tokens": 1,
    "repetition_penalty": 1,
}, headers={
    "Authorization": "Bearer <YOUR_API_KEY>",
    "User-Agent": "<YOUR_APP_NAME>"
})
print(res.json())

Prompt formatting

The following code gives an example of automatically formatting a prompt during inference to match the expected format for the given model. Note, not all models include a prompt_format; make sure you validate that a prompt format is present in your code.

import together

together.api_key = "xxxxx"

model = "garage-bAInd/Platypus2-70B-instruct"

stop_words = list(together.Models.info(model)['config']['stop']) # =
['</s>', '###']
prompt_format = str(together.Models.info(model)['config']['prompt_format'])
# = '### Instruction:\n{prompt}\n### Response:\n'

prompt = "hello"
formatted_prompt = prompt_format.format(prompt=prompt)

for token in together.Complete.create_streaming(prompt=formatted_prompt,
model='upstage/SOLAR-0-70b-16bit', stop=stop_words):
    print(token, end="", flush=True)
print("\n")

Troubleshooting code

You can use RedPajama-INCITE-Chat-7B for troubleshooting your code.

  1. For example, begin the prompt by providing an error message you have.
 I got "ImportError: No module named matplotlib", how to solve it?
  1. Sample response
Make sure that matplotlib is installed on your computer. You can check by running the following command in your terminal:
```
pip install matplotlib
```
If the above command does not work, try the following:
```
sudo apt-get install python-pip
pip install matplotlib
```
  1. Send the prompt to the API with any appropriate parameters. The code below shows an example in Python using the requests package. 
import requests

url = "https://api.together.xyz/inference"

payload = {
    "model": "togethercomputer/togethercomputer/RedPajama-INCITE-7B-Chat",
    "prompt": "I got \"ImportError: No module named matplotlib\", how to solve it?",
    "max_tokens": 128,
    "stop": ".",
    "temperature": 0.1,
    "top_p": 0.7,
    "top_k": 50,
    "repetition_penalty": 1
}
headers = {
    "accept": "application/json",
    "content-type": "application/json",
    "Authorization": "Bearer <YOUR_API_KEY>",
  	"User-Agent": "<YOUR_APP_NAME>",
}    

response = requests.post(url, json=payload, headers=headers)

print(response.text)

Writing Python functions

You can use RedPajama-INCITE-Chat-7B for writing a Python function.

  1. For example, begin the prompt by providing an error message you have.
Generate a Python function to calculate the factorial of a number.
  1. Sample response
Here's a Python function to calculate the factorial of a number:
```
def factorial(n):
    if n == 0:
        return 1
    else:
        return n * factorial(n - 1)
```
This function takes a numeric argument n and returns the factorial of n, which is the product of all integers i such that 0 <= i < n. The base case is when n is 0, which has a factorial of 1. The recursive case calculates the product of all integers i such that 0 <= i < n, including n. n.
  1. Send the prompt to the API with any appropriate parameters. The code below shows an example in Python using the requests package. 
import requests

url = "https://api.together.xyz/inference"

payload = {
    "model": "togethercomputer/togethercomputer/RedPajama-INCITE-7B-Chat",
    "prompt": "Generate a Python function to calculate the factorial of a number.",
    "max_tokens": 256,
    "stop": ".",
    "temperature": 0.1,
    "top_p": 0.7,
    "top_k": 50,
    "repetition_penalty": 1
}
headers = {
    "accept": "application/json",
    "content-type": "application/json",
    "Authorization": "Bearer <YOUR_API_KEY>",
  	"User-Agent": "<YOUR_APP_NAME>",
}    

response = requests.post(url, json=payload, headers=headers)

print(response.text)

HTML Page

You can use RedPajama-INCITE-Chat-7B for generating a simple HTML page.

  1. For example, begin the prompt by providing a task you want to perform in HTML
Generate a simple HTML page showing "Hello world!"
  1. Sample response
Here's an example of a simple HTML page with the text "Hello World!":
```
<!DOCTYPE html>
<html>
  <head>
    <title>Hello World!</title>
  </head>
  <body>
    <h1>Hello World!</h1>
  </body>
</html>
```
  1. Send the prompt to the API with any appropriate parameters. The code below shows an example in Python using the requests package. 
import requests

url = "https://api.together.xyz/inference"

payload = {
    "model": "togethercomputer/togethercomputer/RedPajama-INCITE-7B-Chat",
    "prompt": "Generate a simple HTML page showing \"Hello world!\"",
    "max_tokens": 256,
    "stop": ".",
    "temperature": 0.1,
    "top_p": 0.7,
    "top_k": 50,
    "repetition_penalty": 1
}
headers = {
    "accept": "application/json",
    "content-type": "application/json",
    "Authorization": "Bearer <YOUR_API_KEY>",
  	"User-Agent": "<YOUR_APP_NAME>",
}    

response = requests.post(url, json=payload, headers=headers)

print(response.text)

Grammar correction

Try RedPajama-INCITE-Chat-3B to correct sentences to standard English.

  1. For example, begin the prompt by asking to correct the grammar of a sentence:
Correct this to standard English:
I no Sandwich want.

Correct this to standard English:
If I’m stressed out about something, I tend to have problem to fall asleep.
  1. Sample response
I don't want a sandwich.

If I'm stressed out about something, I tend to have a hard time falling asleep
  1. Send the prompt to the API with any appropriate parameters. The code below shows an example in Python using the requests package. 
import requests

url = "https://api.together.xyz/inference"

payload = {
    "model": "togethercomputer/togethercomputer/RedPajama-INCITE-7B-Chat",
    "prompt": "Correct this to standard English:\nI no Sandwich want.",
    "max_tokens": 256,
    "stop": ".",
    "temperature": 0.1,
    "top_p": 0.7,
    "top_k": 50,
    "repetition_penalty": 1
}
headers = {
    "accept": "application/json",
    "content-type": "application/json",
    "Authorization": "Bearer <YOUR_API_KEY>",
  	"User-Agent": "<YOUR_APP_NAME>",
}    

response = requests.post(url, json=payload, headers=headers)

print(response.text)

Image generation

runwayml/stable-diffusion-v1-5 or any other image model listed in Inference can be used to to generate images.

  1. Make a request using our API
curl https://api.together.xyz/inference \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "User-Agent: YOUR_APP_NAME" \
-d '{"model": "runwayml/stable-diffusion-v1-5", "prompt": "Space robots", "n": 4, "steps": 20 }' \
-o output.json

This request queries the StableDiffusion model to generate images with a prompt of "Space robots". The n parameter specifies how many images the API will return. You should get a response back that resembles the following:

{
  "prompt": ["Space robots"],
  "model": "runwayml/stable-diffusion-v1-5",
  "model_owner": "",
  "tags": {},
  "num_returns": 4,
  "args": {
    "model": "runwayml/stable-diffusion-v1-5n",
    "prompt": "Space robots",
    "n": 4,
    "steps": 20
  },
  "output": {
    "choices": [
      {
        "image_base64": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwc..."
      },
      {
        "image_base64": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwc..."
      },
      {
        "image_base64": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwc..."
      },
      {
        "image_base64": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwc..."
      }
    ],
    "result_type": "image-model-inference"
  },
  "status": "finished",
  "subjobs": []
}

From the above response you can read the output images from the output['choices'][0]['image_base64'].

  1. Now you've generated your first images. You can extract them to JPEG files by running:
cat output.json | jq -r ".output.choices[0].image_base64" | base64 -d > img1.jpg
cat output.json | jq -r ".output.choices[1].image_base64" | base64 -d > img2.jpg
  1. You can also query the API with any HTTP client you like, for example you can do so with Python
import requests

endpoint = 'https://api.together.xyz/inference'

res = requests.post(endpoint, json={
    "model": "runwayml/stable-diffusion-v1-5",
    "prompt": "Space robots",
    "n": 4,
    "steps": 20
}, headers={
    "Authorization": "Bearer <YOUR_API_KEY>",
    "User-Agent": "<YOUR_APP_NAME>"
})

print(res.json())

To query with an input image, try the image_base64 parameter. For best results please send 768x768 JPEG images base64-encoded.

import requests
endpoint = 'https://api.together.xyz/inference'
res = requests.post(endpoint, json={
    "model": "runwayml/stable-diffusion-v1-5",
    "prompt": "Space robots",
    "request_type": "image-model-inference",
    "width": 512,
    "height": 512,
    "steps": 20,
    "n": 4,
    "image_base64": "<IMAGE_BASE64>",
}, headers={ 
    "Authorization": "Bearer <YOUR_API_KEY>",
})
print(res.json())

Image generation with a negative prompt

You can use the Together Inference API to generate images from a text prompt using any supported model. In this tutorial you will learn how to query the Stable Diffusion XL model with both a text prompt and a negative prompt. By the end of this tutorial, you will have a Python script that can query the Together Inference API to generate images along with a negative prompt to tell the model what not to include in the generated image.

  1. Set an environment variable called TOGETHER_API_KEY to your API key in your user settings.
  2. Create a text file named "generate_image.py"
  3. Copy the following text into the file.
import base64
import os

import requests                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                     
url = "https://api.together.xyz/inference"                                                                                                                                                                                                           
model = "stabilityai/stable-diffusion-xl-base-1.0"                                                                                                                                                                                                                  
prompt = "Tokyo crossing"
negative_prompt = "people"
                                                                                                                                                                                                                                                     
print(f"Model: {model}")                                                                                                                                                                                                                             
print(f"Prompt: {repr(prompt)}")                                                                                                                                                                                                                     
print(f"Negative Prompt: {repr(negative_prompt)}")
                                                                                                                                                                                                                                                     
payload = {                                                                                                                                                                                                                                          
    "model": model,                                                                                                                                                                                                                                  
    "prompt": prompt,                                                                                                                                                                                                                                
    "results": 2,                                                                                                                                                                                                                               
    "width": 1024,
    "height": 1024,
    "steps": 20,
    "seed": 42,
    "negative_prompt": negative_prompt,
}                                                                                                                                                                                                                                                    
headers = {                                                                                                                                                                                                                                          
    "accept": "application/json",                                                                                                                                                                                                                    
    "content-type": "application/json",                                                                                                                                                                                                              
    "Authorization": f"Bearer {os.environ['TOGETHER_API_KEY']}"                                                                                                                                                                                      
}                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                     
response = requests.post(url, json=payload, headers=headers, stream=True)                                                                                                                                                                            
response.raise_for_status()

response_json = response.json()
                                                                                                                                                                                                                                                     
# save the first image
image = response_json["output"]["choices"][0]
with open("tokyocrossing.png", "wb") as f:
    f.write(base64.b64decode(image["image_base64"]))
  1. Run the script with:
python3 generate_image.py
  • Replace python3 with the appropriate python interpreter for your platform

The output will be written to a file called tokyocrossing.png in the current working directory.

With Stable Diffusion XL 1.0, you should have just generated an image of a crosswalk in Tokyo without any people in it.

Further reading

Prompt engineering is a relatively new discipline for developing and optimizing prompts to efficiently use language models (LMs) for a wide variety of applications and research topics. The Prompt Engineering Guide is a great introduction to the subject.

Updated 2 months ago