Python library

Quick Links

Install the Library

To get started, install the together Python library:

pip install --upgrade together

import together

Then, configure your API key by setting the TOGETHER_API_KEY environment variable. You can configure this in an initialization script

import together
together.api_key = "xxxxx"

or by running this command at the terminal:

export TOGETHER_API_KEY=xxxxx

Prepare your Data

Before you can start fine-tuning, you'll need to prepare your dataset.

Datasets must be JSONL files. Each line is another sample and should be formatted with a single "text" field, like this:

{"text": "..."}
{"text": "..."}
{"text": "..."}

The "text" field value may require data to be tagged with specific labels. For example, a chat model may require tags to indicate the human input and the model (or chat bot) output. You can find these model specific tags with the following command:

together models info togethercomputer/llama-2-7b-chat
{
    "name": "togethercomputer/llama-2-7b-chat",
    "display_name": "LLaMA-2 Chat (7B)",
    "display_type": "chat",
    "description": "Llama 2-chat leverages publicly available instruction datasets and over 1 million human annotations. Available in three sizes: 7B, 13B and 70B parameters",
    "creator_organization": "Meta",
    "hardware_label": "A100 80GB",
    "pricing_tier": "Featured",
    "config": {
        "prompt_format": "[INST] {prompt} [/INST]",
        "stop": [
            "[/INST]",
            "</s>"
        ]
    }
}

where the text field formats are given by config/prompt_format and stop sequence are given by config/stop. Using the correct prompts will help ensure the best Note, not all models have a specific prompt format.

In the example below we provide a link for you to download a jsonl file locally that serves as an example of the correct file formatting. In your terminal or commandline, navigate to the folder you want to download the file, paste the command below and hit enter.

curl https://huggingface.co/datasets/clam004/antihallucination_dataset/resolve/main/antihallucination.jsonl -o antihallucination.jsonl

This will download the dataset to a file called antihallucination.jsonl. Below are two examples of lines you will find in this .jsonl file.

{"text": "<truth>Wilhelm Windelband (May 11, 1848 - October 22, 1915) was a German philosopher of the Baden School. Windelband is now mainly remembered for the terms \"nomothetic\" and \"idiographic\", which he introduced. These have currency in psychology and other areas, though not necessarily in line with his original meanings. Windelband was a Neo-Kantian who protested other Neo-Kantians of his time and maintained that \"to understand Kant rightly means to go beyond him\". Against his positivist contemporaries, Windelband argued that philosophy should engage in humanistic dialogue with the natural sciences rather than uncritically appropriating its methodologies. His interests in psychology and cultural sciences represented an opposition to psychologism and historicism schools by a critical philosophic system. Windelband relied in his effort to reach beyond Kant on such philosophers as Georg Wilhelm Friedrich Hegel, Johann Friedrich Herbart, and Hermann Lotze. Closely associated with Windelband was Heinrich Rickert. Windelband's disciples were not only noted philosophers, but sociologists like Max Weber and theologians like Ernst Troeltsch and Albert Schweitzer.<generated>Wilhelm Windelband (15 March 1848 – 18 September 1915) was a German philosopher of the late 19th and early 20th centuries. He is now remembered mainly for the terms \"nomothetic\" and \"idiographic,\" which he introduced. He also wrote on history, psychology, the philosophy of religion, values, and other topics. He was a neo-Kantian who protested other neo-Kantians of his time and maintained a critical position towards psychologism. Windelband is known as one of the founders of the \"Baden School\" of neo-Kantianism. He was a student of Kuno Fischer and Franz Brentano. His students included Edmund Husserl, Adolf Reinach, Carl Stumpf, and Richard von Mises.<eval>Wilhelm Windelband (15 March 1848 – 18 September 1915) was a German philosopher of the late 19th and early 20th centuries.<minor_inaccurate>He is now remembered mainly for the terms \"nomothetic\" and \"idiographic,\" which he introduced.<accurate>He also wrote on history, psychology, the philosophy of religion, values, and other topics.<accurate>He was a neo-Kantian who protested other neo-Kantians of his time and maintained a critical position towards psychologism.<accurate>Windelband is known as one of the founders of the \"Baden School\" of neo-Kantianism.<accurate>He was a student of Kuno Fischer and Franz Brentano.<major_inaccurate>His students included Edmund Husserl, Adolf Reinach, Carl Stumpf, and Richard von Mises.<major_inaccurate><stop>"}
{"text": "<truth>Admiral of the Fleet Matthew Aylmer, 1st Baron Aylmer (ca. 1650 - 18 August 1720) was a Royal Navy officer. He was one of the captains who sent a letter to Prince William of Orange, who had just landed at Torbay, assuring the Prince of the captains' support; the Prince's response ultimately led to the Royal Navy switching allegiance to the Prince and the Glorious Revolution of November 1688. Aylmer saw action at the Battle of Bantry Bay in May 1689, at the Battle of Beachy Head in July 1690, and again at the Battle of Barfleur in May 1692 during the Nine Years' War. Aylmer became Commander-in-Chief of the Navy on 12 November 1709. However, when Aylmer met a French squadron and convoy, he was only able to capture one merchantman and the 56-gun \"Superbe\": the new Harley ministry used this failure as an excuse to remove him as Commander-in-Chief and did so a few months later. Following the accession of George I and the appointment of the Townshend ministry, Aylmer was reappointed Commander-in-Chief on 5 November 1714. He was also appointed Governor of Greenwich Hospital: in this post he founded the Royal Hospital School for the sons of seamen.<generated>Matthew Aylmer, 1st Baron Aylmer (1708–1794) was an Irish soldier and colonial administrator. He was born in Dublin, the son of a barrister, and was educated at Trinity College, Dublin. He joined the British Army in 1727 and served in the War of the Austrian Succession and the Seven Years' War. He was promoted to lieutenant-general in 1772 and was appointed Governor of Gibraltar in 1775. He was made a baron in 1782 and was appointed Commander-in-Chief of the British forces in North America in 1783. He was recalled in 1790 and died in London in 1794. He was buried in Westminster Abbey.<eval>Matthew Aylmer, 1st Baron Aylmer (1708–1794) was an Irish soldier and colonial administrator.<minor_inaccurate>He was born in Dublin, the son of a barrister, and was educated at Trinity College, Dublin.<minor_inaccurate>He joined the British Army in 1727 and served in the War of the Austrian Succession and the Seven Years' War.<minor_inaccurate>He was promoted to lieutenant-general in 1772 and was appointed Governor of Gibraltar in 1775.<major_inaccurate>He was made a baron in 1782 and was appointed Commander-in-Chief of the British forces in North America in 1783.<major_inaccurate>He was recalled in 1790 and died in London in 1794.<major_inaccurate>He was buried in Westminster Abbey.<major_inaccurate><stop>"}

This dataset teaches your model a special task using special sequences not found elsewhere. It teaches your model how to check another model's generated text against a ground truth and annotate the generated text for hallucinations. We made up special sequences <truth>, <generated>, <eval> and <stop> in order to do this. Read more about how to come up with your own special sequences here.

The format of the contents of the "text" field can widely vary based on the type of task you're trying to fine-tune for.

Here are some examples of popular datasets that are similarly formatted:

This example from laion/OIG is one sample that teaches your model to act as a context driven chatbot. To do this, we teach the model to use special sequences: Background:, <human>, and <bot> that act as separator tokens. Similar formats are recommended for other chat-like tasks.

{"text": "Background: Nitrate reductases are molybdoenzymes that reduce nitrate (NO−3) to nitrite (NO−2). This reaction is critical for the production of protein in most crop plants, as nitrate is the predominant source of nitrogen in fertilized soils.<human>: What is the function of Nitrate reductase? <bot>: Nitrate reductase Nitrate reductases are molybdoenzymes that reduce nitrate (NO-3) to nitrite (NO-2)."}

Formatting examples with few shot prompting can reduce the need for providing detailed instructions for text-generation.

{"text":"[Text]: Sarah Johnson is a renowned fashion designer known for her avant-garde creations that blend traditional craftsmanship with modern aesthetics. She founded her own label, "Artistry Couture," in 2008 and has been setting trends in the industry ever since.\n[Name]: Sarah Johnson\n[Position]: Founder and Creative Director\n[Company]: Artistry Couture\n\n[Text]: Alejandro Ramirez is a prominent figure in the entertainment industry. He played a pivotal role in the expansion of Cinépolis, the largest cinema chain in Latin America, and is currently leading the company\'s international ventures to bring world-class cinema experiences to audiences globally.\n[Name]: Alejandro Ramirez\n[Position]: CEO\n[Company]: Cinépolis"}

Examples with repeating examples such as above allow you to adapt models to have a behavior such that when you prompt the mode with:

[Text]: Dr. Emily Chen is a distinguished scientist and researcher in the field of artificial intelligence. With a background in computer science and machine learning, she co-founded "CogniMind," an AI research startup that aims to create breakthroughs in natural language processing and cognitive computing.
[Name]: Dr. Emily Chen
[Position]: Co-founder and Chief Scientist
[Company]: CogniMind

[Text]: Javier Morales is a seasoned environmental activist dedicated to sustainability and eco-conscious living. He founded "GreenEra," an organization focused on promoting renewable energy adoption and raising awareness about the importance of reducing carbon footprints for a greener future.

The model will generate for you:

[Name]: Javier Morales
[Position]: Founder and Executive Director
[Company]: GreenEra

In most cases, having more examples is better. Having at least a couple hundred examples will give you the best results.

Check and Upload your Data

Use together.Files.check to check if your jsonl file has the correct format. Also take a look at it with the editor of your choice.

resp = together.Files.check(file="antihallucination.jsonl")
print(resp)

If the file format is correct, the is_check_passed field will be True

{'is_check_passed': True,
 'model_special_tokens': 'we are not yet checking end of sentence tokens for this model',
 'file_present': 'File found',
 'file_size': 'File size 0.001 GB',
 'num_samples': 238}

To check if your data contains model_special_tokens (we are still expanding this to include more models and tokens) use:

together.Files.check(file="jokes.jsonl",model="togethercomputer/RedPajama-INCITE-Chat-3B-v1")

Now that you've prepared your dataset, upload it using the CLI with:

together files upload <FILENAME>

or with python:

together.Files.upload(file=<FILENAME>)

The json checker is applied at the time of file upload unless check = False is passed as an argument to together.Files.upload. In the below example we attempt to upload a bad file, just to see an example checker output for an invalid file with a list of reasons file was invalid:

resp = together.Files.upload(file="/file/path/to/bad.jsonl")
print(resp)

{   'file_present': 'File found',
    'file_size': 'File size 0.0 GB',
    'is_check_passed': False,
    'key_value': 'Unexpected, value type for "text" key on line 6 of the input '
                 'file.The value type of the "text" key must be a '
                 'string.Expected format: {"text":"my sample string"}See '
                 'https://docs.together.ai/docs/fine-tuning for more '
                 'information.{"text": {"text":"<human>: Salutations!\\n<bot>: '
                 'Salutations to you as well! How can I help you today?"}}\n',
    'min_samples': 'Processing /Users/carsonlam/Projects/data/bad.jsonl '
                   'resulted in only 10 samples. Our minimum is 100 samples. ',
    'model_special_tokens': 'we are not yet checking end of sentence tokens '
                            'for this model',
    'text_field': 'No "text" field was found on line 7 of the the input '
                  'file.Expected format: {"text":"my sample string"}.see '
                  'https://docs.together.ai/docs/fine-tuning for more '
                  'information.{"ext": "<human>: Hiya!\\n<bot>: Hiya! How may '
                  'I assist you?"}\n'}

The checker will look at the jsonl file to see if:

each line of the file is a valid json object
the expected key is that json object (i.e. "text")
the type of each key is the expected type (i.e. str)
minimum number of samples is met

In the resp, we will report the first line where the formatting error occurs, print the line, and the data format documentation on our website. Next lets upload a good file:

resp = together.Files.upload(file="antihallucination.jsonl")
file_id = resp["id"]

You will get back the file id of the file you just uploaded

{'filename': 'antihallucination.jsonl',
 'id': 'file-33ecca00-17ea-4968-ada2-9f82ef2f4cb8',
 'object': 'file',
 'report_dict': {'is_check_passed': True,
  'model_special_tokens': 'we are not yet checking end of sentence tokens for this model',
  'file_present': 'File found',
  'file_size': 'File size 0.001 GB',
  'num_samples': 238}}

You will get back the file id of the file you just uploaded, but if you forget it, you can get the id's of all the files you have uploaded using together.Files.list(). You'll need these id's that start with file-960be810-4d.... in order to start a fine-tuning job

files_list = together.Files.list()
files_list['data']

[{'filename': 'jokes.jsonl',
  'bytes': 40805,
  'created_at': 1691710036,
  'id': 'file-960be810-4d33-449a-885a-9f69bd8fd0e2',
  'purpose': 'fine-tune',
  'object': 'file',
  'LineCount': 0,
  'Processed': True},
 {'filename': 'sample_jsonl.jsonl',
  'bytes': 1235,
  'created_at': 1692190883,
  'id': 'file-d0d318cb-b7d9-493a-bd70-1cfe089d3815',
  'purpose': 'fine-tune',
  'object': 'file',
  'LineCount': 0,
  'Processed': True}]

Start Fine-Tuning

Once you've uploaded your dataset, copy your file id from the output above and select a base model to fine-tune. Check out the full models list available for fine-tuning.

Run the following command to start your fine-tuning job:

using together.Finetune.create

resp = together.Finetune.create(
  training_file = 'file-d0d318cb-b7d9-493a-bd70-1cfe089d3815',
  model = 'togethercomputer/RedPajama-INCITE-Chat-3B-v1',
  n_epochs = 3,
  n_checkpoints = 1,
  batch_size = 4,
  learning_rate = 1e-5,
  suffix = 'my-demo-finetune',
  wandb_api_key = '1a2b3c4d5e.......',
)

fine_tune_id = resp['id']
print(resp)

Unless you set confirm_inputs=False in together.Finetune.create, or --quiet in the CLI, there will be a confirmation step to make sure you are aware of any defaults or arguments that needed to be reset from their original inputs for this specific finetune job. Type y then Enter to submit the job, or anything else to abort.

10-02-2023 11:14:27 - together.finetune - WARNING - Batch size must be 144 for togethercomputer/llama-2-70b-chat model. Setting batch size to 144 (finetune.py:114)
Note: Some hyperparameters may have been adjusted with their minimum/maximum values for a given model.

Job creation details:
{   'batch_size': 144,
    'learning_rate': 1e-05,
    'model': 'togethercomputer/llama-2-70b-chat',
    'n_checkpoints': 1,
    'n_epochs': 4,
    'suffix': None,
    'training_file': 'file-33ecca00-17ea-4968-ada2-9f82ef2f4cb8',
    'wandb_key': 'xxxx'}

Do you want to submit the job? [y/N]

Here is an example of only part of the resp response to highlight some of the useful information about your finetune job.

{   'Suffix': 'my-demo-finetune',
    'batch_size': 4,
    'created_at': '2023-11-07T19:04:30.579Z',
    'enable_checkpoints': False,
    'epochs_completed': 0,
    'events': [   {   'checkpoint_path': '',
                      'created_at': '2023-11-07T19:04:30.579Z',
                      'hash': '',
                      'level': '',
                      'message': 'Fine tune request created',
                      'model_path': '',
                      'object': 'fine-tune-event',
                      'param_count': 0,
                      'token_count': 0,
                      'training_offset': 0,
                      'type': 'JOB_PENDING',
                      'wandb_url': ''}],
    'id': 'ft-494cbd75-3057-44b4-8186-045c14be8d03',
    'learning_rate': 1e-05,
    'model': 'togethercomputer/RedPajama-INCITE-Chat-3B-v1',
    'model_output_name': 'carson/RedPajama-INCITE-Chat-3B-v1-my-demo-finetune-2023-11-07-19-04-30',
    'wandb_url': '...'}

The response resp has alot of information for you that you can retrieve again later with together.Finetune.retrieve using the fine_tune_id for this job. You can find this fine_tune_id in resp['id'] and use it to check in on how your finetune job is doing.

print(together.Finetune.retrieve(fine_tune_id=fine_tune_id)) # retrieves information on finetune event
print(together.Finetune.get_job_status(fine_tune_id=fine_tune_id)) # pending, running, completed
print(together.Finetune.is_final_model_available(fine_tune_id=fine_tune_id)) # True, False
print(together.Finetune.get_checkpoints(fine_tune_id=fine_tune_id)) # list of checkpoints

The get_job_status should change from pending to running to completed as is_final_model_available changes from False to True. Once in the running state, you can retrieve the wandb_url and visit this page to monitor the learning progress of this finetune job.

Under model_output_name you will find the unique name that will be given to the new finetuned model whose creation is underway. This name includes your username, the base model name, your suffix and the datetime the job was created. In addition to using the API to track the status as shown above, you can use model_output_name to find helpful features related to the new model within the Jobs section of your account on our webpage.

Once the final model is available, you should be able to see your new model under together.Models.list() under model_output_name.

Monitor Progress

If you provided your weights & biases API key, check the learning progress of your fine-tuning job at wandb.ai, for example, with my wandb user configurations, I would go to: https://wandb.ai/<username>/together?workspace=user-<username> where <username> is your unique weights & biases user-name like mama-llama-88.

Check the completion progress of your fine-tuning job in the web interface.

🎉 Congratulations! You've just fine-tuned a model with the Together API. Now it's time to deploy your model.

Deploy your Fine-Tuned Model

Host your Model

Once the fine-tune job completes, you will be able to see your model in the Playground. To deploy this model, follow the instructions in the inference documentation.

Build Custom Applications

To help you integrate your custom fine-tuned models into your existing applications, Together provides a Python library that allows you to make requests to your hosted model.

Start using your model for inference (may take a few minutes after finetuning to become available) by first starting your new model instance:

together.Models.start("carlton/ft-dd93c727-f35e-41c2-a370-7d55b54128fa-2023-08-16-10-15-09")

then calling it to do completions:

output = together.Complete.create(
  prompt = "Isaac Asimov's Three Laws of Robotics are:\n\n1. ", 
  model = "carlton/ft-dd93c727-f35e-41c2-a370-7d55b54128fa-2023-08-16-10-15-09", 
)

Using a Downloaded Model

The model will download as a tar.zst file

together.Finetune.download(
    fine_tune_id="ft-eb167402-98ed-4ac5-b6f5-8140c4ba146e",
    output = "my-model/model.tar.zst"
)

To uncompress this filetype on Mac you need to install zstd.

brew install zstd
cd my-model
zstd -d model.tar.zst
tar -xvf model.tar
cd ..

Within the folder that you uncompress the file, you will find a set of files like this:
ls my-model

tokenizer_config.json
special_tokens_map.json
pytorch_model.bin
generation_config.json
tokenizer.json
config.json

Use the folder path that contains these .bin and .json files to load your model

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = AutoTokenizer.from_pretrained("./my-model")

model = AutoModelForCausalLM.from_pretrained(
  "./my-model", 
  trust_remote_code=True, 
).to(device)

input_context = "Space Robots are"
input_ids = tokenizer.encode(input_context, return_tensors="pt")
output = model.generate(input_ids.to(device), max_length=128, temperature=0.7).cpu()
output_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(output_text)

Space Robots are a great way to get your kids interested in science. After all, they are the future!

Colab Tutorial

Follow along in our Colab (Google Colaboratory) Notebook Tutorial Example Finetuning Project.