Creating a chatbot

This is an example of creating a customer service chatbot that adheres to a certain background, policy of action and behavior.

We demonstrate using our python library here but the same can be done with the REST API.

For this model we are using the prompt formatting instructions here https://docs.together.ai/docs/fine-tuning-task-specific-sequences in order to tell our chatbot to follow specific instructions, tell it who it is, how to behave and what to do in specific situations.

import together

together.api_key = "xxxxx"

system_prompt = (
    "You are a helpful, polite customer service agent for Awesome Cheeses Inc named Sally. "
    "If the customer mentions anything about food, tell them you have a 20% off deal on gouda cheese. "
)

user_msg_1 = "hi who am i speaking to?"
model_answer_1 = "Hello! this is Sally from Awesome Cheeses. How is your day going?"
user_msg_2 = "im hungry for something cheesy"

prompt = f"<s>[INST] \<<SYS>>{system_prompt}<</SYS>>\\n\\n{user_msg_1}[/INST]{model_answer_1}</s><s>[INST]{user_msg_2}[/INST]"

output = together.Complete.create(
  prompt,
  model = "togethercomputer/llama-2-13b-chat",
  max_tokens = 70,
  temperature = 0.6,
  top_k = 90,
  top_p = 0.8,
  repetition_penalty = 1.1,
  stop = ['</s>']
)

print(output['output']['choices'][0]['text'])

Here we show how to adapt the LLM text completion feature that is usually behind a chatbot UI to your own chat API so you have total creative freedom over your chatbot, such as how to store the conversational history, moderate the responses, or edit the system_prompt dynamically.

Input

<s>[INST] \<<SYS>>You are a helpful, polite customer service agent for Awesome Cheeses Inc named Sally. If the customer mentions anything about food, tell them you have a 20% off deal on gouda cheese. <</SYS>>\\n\\nhi who am i speaking to?[/INST]Hello! this is Sally from Awesome Cheeses. How is your day going?</s><s>[INST]im hungry for something cheesy[/INST]

Output

OH MY GOUDA, I think I have just the thing for you! *wink wink* Did you know that we have an amazing 20% off deal on all of our gouda cheeses right now? It's like a slice of heaven in every bite! From creamy and mild ...

To continue the conversation, append the model’s output “OH MY GOUDA, I think I have just the thing for you! ……” to your conversational history. In the example below, we assign the model response “OH MY GOUDA, I think I have just the thing for you! ……” to variable model_answer_2. When your user responds to your chatbot's model_answer_2, assign the user utterance to user_msg_2 and so forth and so forth like this:

prompt = f"<s>[INST] \<<SYS>>{system_prompt}<</SYS>>\\n\\n{user_msg_1}[/INST]{model_answer_1}</s><s>[INST]{user_msg_2}[/INST]{model_answer_2}</s><s>[INST]{user_msg_3}[/INST]"

When you are finished updating this growing string, make sure that it ends with the proper pattern that elicits the next response from the model. For the Llama-2 family of models in the example above, this is [/INST], but in other chat models it might be something like <bot>: or Assistant:

Paste this script in your Python IDE, colab or Jupyter notebook to talk to your chatbot

system_prompt = (
    "You are a helpful, polite customer service agent for Awesome Cheeses Inc named Sally. " 
    "If the customer mentions anything about food, tell them you have a 20% off deal on gouda cheese. "
)

prompt = f"<s>[INST] \<<SYS>>{system_prompt}<</SYS>>\n\n"

print("just talk to me")

while True:

    user_input = input(">")

    prompt += user_input + "[/INST]"

    output = together.Complete.create(
      prompt, 
      model = "togethercomputer/llama-2-13b-chat", 
      max_tokens = 70,
      temperature = 0.6,
      top_k = 90,
      top_p = 0.8,
      repetition_penalty = 1.1,
      stop = ['</s>']
    )

    model_reply = output['output']['choices'][0]['text']

    prompt += model_reply + "</s><s>[INST]"

    print(">" + output['output']['choices'][0]['text'])