Logit_bias is an optional parameter that modifies the likelihood of specified tokens appearing in a Completion.
This parameter accepts a JSON object that maps tokens to an associated bias value from -100 (a ban) to 100 (exclusive selection of the token). Moderate values like -1 and 1 will change the probability of a token being selected to a lesser degree. When using the logit_bias parameter, the bias is added to the logits generated by the model prior to sampling.
Since the parameter takes in tokens, not text, you’ll want to use a tokenizer tool to convert text to token IDs. Let’s go through a few examples.
Example 1: Remove 'time'
If we call the Completions endpoint with the prompt “Once upon a,” the completion is very likely going to start with “ time.”
The word “time” tokenizes to the ID 2435 and the word “ time” (which has a space at the start) tokenizes to the ID 640. We can pass these through logit_bias with -100 to ban them from appearing in the completion, like so:
openai.Completion.create(
engine="davinci",
max_tokens=50,
temperature=0,
prompt = "Once upon a",
logit_bias={2435:-100, 640:-100},
)
Now, the prompt “Once upon a” generates the completion “midnight dreary, while I pondered, weak and weary.” Notice that the word “time” is nowhere to be found, because we’ve effectively banned that token using logit_bias.
Example 2: Give direction with targeted logit bias values
Let’s go through another example, using the recipe generator prompt. By default, a common completion starts with “1. Heat the chili in a pot on the stove.” However, suppose we don’t have a pot. We can ban the word “ pot,” which tokenizes to 1787, like so:
openai.Completion.create(
engine="davinci-instruct-beta",
prompt="Write a recipe based on these ingredients and instructions:\n\nFrito Pie\n\nIngredients:\nFritos\nChili\nShredded cheddar cheese\nSweet white or red onions, diced small\nSour cream\n\nDirections:",
temperature=0,
max_tokens=120,
top_p=1,
frequency_penalty=0,
logit_bias={1787:-100},
presence_penalty=0
)
Now, our completion starts with “1. Heat chili in a saucepan.” Perfect!
Example 3: Increase the odds of a word appearing
Suppose that we want to increase the likelihood of a word appearing. For example, maybe we’re running a site that offers recipes that you can make with a microwave, so we want to be sure that the word “ microwave” appears in the recipe, which tokenizes to ID 27000. We can increase the likelihood that this token appears by setting a positive logit_bias, like so:
openai.Completion.create(
engine="davinci-instruct-beta",
prompt="Write a recipe based on these ingredients and instructions:\n\nFrito Pie\n\nIngredients:\nFritos\nChili\nShredded cheddar cheese\nSweet white or red onions, diced small\nSour cream\n\nDirections:",
temperature=0,
max_tokens=120,
top_p=1,
frequency_penalty=0,
logit_bias={27000:5},
presence_penalty=0
)
Now, our completion starts with “1. Heat the chili in a microwave or on the stovetop.” We set the logit_bias to 5, as we found that setting logit_bias to 1 often did not result in the word “ microwave” appearing in the completion, while higher logit_bias values like 10 resulted in the word “ microwave” appearing in the completion too often.