Kılavuzlar

Toplu API

Process large volumes of requests asynchronously at 50% lower cost. Ideal for workloads that don't require real-time responses.

50%
cost savings vs standard API
24h
completion window
50K
max requests per batch
All
models supported
Note: Batch API supports all models that support the Responses API.

When to use Batch API

Choose between synchronous and asynchronous processing based on your latency requirements and cost budget.

Attribute Standard API (Sync) Batch API (Async)
Latency Seconds (real-time) Up to 24 hours
Cost Standard pricing 50% discount
Rate limits Standard rate limits apply Separate, higher limits
Max requests Unlimited (rate-limited) 50,000 per batch
Best for Chatbots, real-time apps, interactive tools Evaluations, embeddings, bulk classification, data enrichment
Response delivery Streaming or blocking Downloadable JSONL file

Creating a batch

Batch requests are submitted as a .jsonl file where each line is a separate API request. Upload the file, then create the batch.

Step 1: Prepare your JSONL file

Each line must be a JSON object with a custom_id, method, url, and body.

requests.jsonl 
{"custom_id": "req-1", "method": "POST", "url": "/v1/responses", "body": {"model": "glomaxgpt-ultra", "input": [{"role": "user", "content": "Summarize the history of the Roman Empire."}], "max_output_tokens": 500}}
{"custom_id": "req-2", "method": "POST", "url": "/v1/responses", "body": {"model": "glomaxgpt-ultra", "input": [{"role": "user", "content": "Explain quantum entanglement in simple terms."}], "max_output_tokens": 300}}
{"custom_id": "req-3", "method": "POST", "url": "/v1/responses", "body": {"model": "glomaxgpt-ultra", "input": [{"role": "user", "content": "Write a haiku about autumn."}], "max_output_tokens": 100}}

Step 2: Upload and create batch

create-batch.py 
from GlomaxGPT import GlomaxGPT

client = GlomaxGPT()

# Step 1: Upload the JSONL file
with open("requests.jsonl", "rb") as f:
    batch_file = client.files.create(
        file=f,
        purpose="batch"
    )

print(f"Uploaded file: {batch_file.id}")

# Step 2: Create the batch
batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/responses",
    completion_window="24h",
    metadata={
        "description": "Nightly summarization job"
    }
)

print(f"Batch created: {batch.id}")
print(f"Status: {batch.status}")
create-batch.js 
import GlomaxGPT from "GlomaxGPT";
import fs from "fs";

const client = new GlomaxGPT();

// Step 1: Upload the JSONL file
const batchFile = await client.files.create({
  file: fs.createReadStream("requests.jsonl"),
  purpose: "batch"
});

console.log(`Uploaded file: ${batchFile.id}`);

// Step 2: Create the batch
const batch = await client.batches.create({
  input_file_id: batchFile.id,
  endpoint: "/v1/responses",
  completion_window: "24h",
  metadata: {
    description: "Nightly summarization job"
  }
});

console.log(`Batch created: ${batch.id}`);
console.log(`Status: ${batch.status}`);

Checking batch status

Poll the batch status endpoint to track progress. A batch moves through several states before completion.

validating

The batch file is being validated for formatting errors.

in_progress

Requests are actively being processed.

completed

All requests have been processed. Results are available.

failed

The batch failed to process. Check the error file.

expired

The batch was not completed within the 24-hour window.

cancelled

The batch was cancelled before completion.

Warning: Batch requests expire after 24 hours if not completed. Design your polling logic to handle the expired status and retry if needed.
poll-batch.py 
import time

def wait_for_batch(batch_id, poll_interval=60):
    while True:
        batch = client.batches.retrieve(batch_id)
        status = batch.status

        print(f"Status: {status} | "
              f"Completed: {batch.request_counts.completed}/"
              f"{batch.request_counts.total}")

        if status in ("completed", "failed", "expired", "cancelled"):
            return batch

        time.sleep(poll_interval)

batch = wait_for_batch("batch_abc123")
print(f"Final status: {batch.status}")
poll-batch.js 
async function waitForBatch(batchId, pollInterval = 60000) {
  const terminalStates = ["completed", "failed", "expired", "cancelled"];

  while (true) {
    const batch = await client.batches.retrieve(batchId);
    const { status, request_counts } = batch;

    console.log(`Status: ${status} | Completed: ${request_counts.completed}/${request_counts.total}`);

    if (terminalStates.includes(status)) return batch;

    await new Promise(resolve => setTimeout(resolve, pollInterval));
  }
}

const batch = await waitForBatch("batch_abc123");
console.log(`Final status: ${batch.status}`);

Retrieving results

Once the batch is complete, download the output file. Results are returned as a JSONL file where each line corresponds to a request, identified by custom_id.

retrieve-results.py 
import json

# Download the output file
output_file = client.files.content(batch.output_file_id)
results_text = output_file.text

# Parse each result line
results = {}
for line in results_text.strip().split("\n"):
    result = json.loads(line)
    custom_id = result["custom_id"]
    response_body = result["response"]["body"]
    output_text = response_body["output"][0]["content"][0]["text"]
    results[custom_id] = output_text

print(results["req-1"])  # Roman Empire summary
print(results["req-2"])  # Quantum entanglement explanation
retrieve-results.js 
// Download the output file
const outputFile = await client.files.content(batch.output_file_id);
const resultsText = await outputFile.text();

// Parse each result line
const results = {};
for (const line of resultsText.trim().split("\n")) {
  const result = JSON.parse(line);
  const { custom_id } = result;
  const outputText = result.response.body.output[0].content[0].text;
  results[custom_id] = outputText;
}

console.log(results["req-1"]); // Roman Empire summary

Error handling

Individual requests within a batch can fail independently. Failed requests appear in a separate error output file. The batch itself may still be marked completed even if some requests failed.

error-handling.py 
if batch.error_file_id:
    error_file = client.files.content(batch.error_file_id)
    for line in error_file.text.strip().split("\n"):
        error = json.loads(line)
        print(f"Failed request: {error['custom_id']}")
        print(f"  Error code: {error['error']['code']}")
        print(f"  Message: {error['error']['message']}")

# Check request counts
counts = batch.request_counts
print(f"Total: {counts.total}")
print(f"Completed: {counts.completed}")
print(f"Failed: {counts.failed}")

Limits & quotas

The Batch API has specific limits designed to balance performance and resource usage across all users.

Limit Value Notes
Max requests per batch 50,000 Split larger workloads into multiple batches
Max file size 200 MB Applies to the input JSONL file
Completion window 24 hours Batches expire after 24h if not finished
Concurrent batches Org-level limit Varies by usage tier
Enqueued token limit Tier-based See usage tier documentation

Cancelling a batch

You can cancel an in-progress batch. Requests that have already been processed will still be available in the output file.

cancel-batch.py 
client.batches.cancel("batch_abc123")