Learn how to return log probabilities for your output tokens & build better classifiers.
logprobs: 1
to your API call as seen below.
{ "New": -0.39648438 }
. The “New” token has a logprob of -0.39648438, but this isn’t very helpful by itself. However, we can quickly convert it to a probability by taking the exponential of it.
-0.012512207
. After taking the exponential of this, we get a probability of 98.7%. We’re using a small and fast LLM here (llama 3.1 8B) which is great, but using logprobs, we can also tell when the model is unsure of its answer and see if we need to route it to a bigger LLM.
logprobs
to show how to build a more robust classifier (and a cheaper classifier, using a smaller model for most queries but selectively using bigger models when needed). There are many other use cases for logprobs
around autocompletion, keyword selection, and moderation.