How does the Batch API work?
The Batch API endpoint, as documented here, allows users to submit requests for asynchronous batch processing. We will process these requests within 24 hours. The details of each request will be read from a pre-uploaded file, and the responses will be written to an output file. You can query the batch object for status updates and results.
What is the pricing for the Batch API?
Each model will be offered at 50% cost discount vs. the synchronous APIs. Here's more information regarding our API pricing.
What models can I batch?
The following models will be available for batching:
gpt-4o
gpt-4o-mini
gpt-4o-2024-05-13
gpt-3.5-turbo
gpt-3.5-turbo-16k
gpt-4
gpt-4-32k
gpt-4-turbo-preview
gpt-4-vision-preview
gpt-4-turbo
gpt-3.5-turbo-1106
gpt-4-0314
gpt-4-turbo-2024-04-09
gpt-4-32k-0314
gpt-4-32k-0613
text-embedding-3-large
text-embedding-3-small
text-embedding-ada-002
What happens if the API doesn’t complete my request within the promised time?
We aim to get all requests within 24 hours. If a batch expires (i.e. it could not be completed within the SLA time window), then remaining work is cancelled and any already completed work is returned. Developers will be charged for any completed work.
What’s the limit of how many requests I can batch?
Embeddings APIs have a limit of 1 million enqueued requests at a time.
For all other APIs, there is no limit on the number of requests you can batch; however, each usage tier has an associated batch rate limit. Your batch rate limit includes the maximum number of input tokens you have enqueued at one time. You can find your rate limits here.
Can I change the time it takes to hear back from the Batch API?
Our current specified time window is 24 hours. We currently cannot change this time period.
Is streaming supported on the Batch API?
No, streaming is not supported on the Batch API.
Are images supported on the Batch API?
Yes, images are supported on the Batch API.
I got an error message “The URL provided for this request does not prefix-match the batch endpoint”. What should I do?
If you get this error, this means your URL is incorrectly formatted for the Batch API endpoint. Please refer to our documentation here to make sure you’re calling the endpoint correctly.
Does usage of the Batch API count against my other rate limits?
Batch API rate limits are completely separate from existing limits.
What happens if a batch is cancelled?
If a batch is manually cancelled then whatever results have already been completed are returned. Developers will be charged for any completed work.
What happens if a batch expires?
If a batch expires (i.e. it could not be completed within the SLA time window), then remaining work is cancelled and any already completed work is returned. Developers will be charged for any completed work.
What statuses can I expect on batch jobs?
The batch job can have any of the following statuses:
Validating: Validation of the uploaded file is in progress before the batch can begin.
Failed: The file has failed the validation process.
In Progress: The file was successfully validated and the batch process is underway.
Finalizing: The batch job has completed and the results are being prepared.
Completed: The batch job is complete and the results are ready.
Expired: The batch could not be completed within the SLA time window.
Canceling: Cancellation of the batch job has been initiated.
Canceled: The batch job was canceled.
How can I check which batch jobs I've already created?
You can get a list of all your batch jobs using the API call specified here.
Is zero data retention supported on this endpoint?
If you have zero data retention enabled for your org, please note that zero data retention is not supported on this endpoint.