Chat completions
Chat request bodies on a batch line. There is no live POST /v1/chat/completions.
Not a live HTTP endpoint
Do not POST to /v1/chat/completions. Put that path on each JSONL line as url, then submit the
file with POST /v1/batches and "endpoint": "/v1/chat/completions".
The OpenAI Chat Completions request, unchanged, as the body of a batch line.
Anything an OpenAI-compatible client already builds works as-is once you wrap it in that line.
Requires the batches:write scope on the key that creates the batch.
{"custom_id":"ticket_4821","method":"POST","url":"/v1/chat/completions","body":{"model":"llama-3.3-70b","messages":[{"role":"system","content":"You summarize support tickets in two sentences."},{"role":"user","content":"Customer cannot log in after password reset."}],"temperature":0.2,"max_tokens":200},"metadata":{"entity":"user:usr_8421"}}Submit the file:
POST /v1/batches HTTP/1.1
Host: REGION.api.lararouter.com
Authorization: Bearer $LARAROUTER_API_KEY
Content-Type: application/json
{
"input_file_id": "file_9Hs4TnQ2",
"endpoint": "/v1/chat/completions",
"completion_window": "24h"
}Most codebases already have something that produces the body object. Write it as JSON on the line:
$line = [
'custom_id' => "ticket_{$ticket->id}",
'method' => 'POST',
'url' => '/v1/chat/completions',
'body' => [
'model' => 'llama-3.3-70b',
'messages' => [
['role' => 'system', 'content' => 'You summarize support tickets in two sentences.'],
['role' => 'user', 'content' => $ticket->body],
],
'temperature' => 0.2,
'max_tokens' => 200,
],
'metadata' => ['entity' => "user:{$ticket->user_id}"],
];Parameters
Prop
Type
stream and stream_options are rejected at validation. There is nothing to stream to. Results are
written to a file and collected when the job finishes.
Response
Results arrive in the batch's output_file_id, one line per input line, keyed by your custom_id.
The response.body is the completion object an OpenAI client would have received.
{
"custom_id": "ticket_4821",
"response": {
"status_code": 200,
"request_id": "req_01JD8XK4M2",
"body": {
"id": "chatcmpl_01JD8XK4M2",
"object": "chat.completion",
"created": 1754150400,
"model": "llama-3.3-70b",
"choices": [
{
"index": 0,
"message": { "role": "assistant", "content": "The customer is locked out following a password reset…" },
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 284,
"completion_tokens": 96,
"total_tokens": 380,
"cost_usd": 0.00042
}
}
},
"error": null
}usage.cost_usd is a Lararouter addition, and the per-line costs sum to the cost_usd on the batch
object.
finish_reason is stop, length (hit max_tokens), tool_calls, or content_filter.
A line that fails carries response: null and a populated error, and lands in error_file_id
instead. See Batches.
Tool calling
Describe functions in tools and the model may respond with tool_calls instead of prose. In a batch
that answer comes back in the result file rather than to a waiting process, so a tool loop is one
batch per round: read the tool_calls, run the functions locally, then submit a second batch whose
lines carry the tool results appended to the same messages.
{"custom_id":"order_4821","method":"POST","url":"/v1/chat/completions","body":{"model":"llama-3.3-70b","messages":[{"role":"user","content":"Is order 4821 delivered?"}],"tools":[{"type":"function","function":{"name":"get_order_status","description":"Look up the delivery status of an order.","parameters":{"type":"object","properties":{"order_id":{"type":"string"}},"required":["order_id"]}}}]}}The result line carries the call the model wants to make:
{
"custom_id": "order_4821",
"response": {
"status_code": 200,
"body": {
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"tool_calls": [
{
"id": "call_9xQ2",
"type": "function",
"function": { "name": "get_order_status", "arguments": "{\"order_id\":\"4821\"}" }
}
]
},
"finish_reason": "tool_calls"
}
]
}
},
"error": null
}Run the function, then build round two from the same custom_id so the two halves stay matched:
$message = $result['response']['body']['choices'][0]['message'];
$call = $message['tool_calls'][0];
$next = [
'custom_id' => $result['custom_id'],
'method' => 'POST',
'url' => '/v1/chat/completions',
'body' => [
'model' => 'llama-3.3-70b',
'messages' => [
['role' => 'user', 'content' => 'Is order 4821 delivered?'],
$message,
[
'role' => 'tool',
'tool_call_id' => $call['id'],
'content' => Order::findOrFail(json_decode($call['function']['arguments'], true)['order_id'])->status,
],
],
'tools' => $tools,
],
];Each round is a separate batch, and a separate wait
Every round is billed on its own and gets its own audit log entry, so attribute all of them to the same entity or per-user reporting will only count half the exchange. It also means a three-round tool loop can take three completion windows. Deep agentic loops are a poor fit for batch; prefer a single call with everything the model needs already in the prompt.
Structured outputs
Constrain the response to a JSON Schema and the model can't return anything that doesn't validate. This matters more in batch than it does interactively: nobody is watching to catch a malformed answer, and a schema turns "parse 4,200 responses and hope" into a guarantee.
{"custom_id":"ticket_4821","method":"POST","url":"/v1/chat/completions","body":{"model":"llama-3.3-70b","messages":[{"role":"user","content":"Classify: the checkout page crashes on Safari."}],"response_format":{"type":"json_schema","json_schema":{"name":"ticket_classification","strict":true,"schema":{"type":"object","properties":{"category":{"type":"string","enum":["bug","billing","feature","other"]},"severity":{"type":"integer","minimum":1,"maximum":5}},"required":["category","severity"],"additionalProperties":false}}}}}Put the schema on the line as response_format:
$line = [
'custom_id' => "ticket_{$ticket->id}",
'method' => 'POST',
'url' => '/v1/chat/completions',
'body' => [
'model' => 'llama-3.3-70b',
'messages' => [['role' => 'user', 'content' => $ticket->body]],
'response_format' => [
'type' => 'json_schema',
'json_schema' => [
'name' => 'ticket_classification',
'strict' => true,
'schema' => [
'type' => 'object',
'properties' => [
'category' => ['type' => 'string', 'enum' => ['bug', 'billing', 'feature', 'other']],
'severity' => ['type' => 'integer', 'minimum' => 1, 'maximum' => 5],
],
'required' => ['category', 'severity'],
'additionalProperties' => false,
],
],
],
],
];Not every model supports strict schema enforcement. Check supports.structured_output on
GET /v1/models/{model} before relying on it; models without it fall back to
best-effort JSON, which can fail to parse. In a batch that failure shows up hours later.
Grouping related lines
Give lines the same conversation in their metadata when they belong to one thread. It is stored on
the audit record and works across batches, so a multi-turn exchange assembled
over several nightly jobs still groups together.
{"custom_id":"ticket_4821_turn2","method":"POST","url":"/v1/chat/completions","body":{"model":"llama-3.3-70b","messages":[{"role":"user","content":"And the refund policy?"}]},"metadata":{"entity":"user:usr_8421","conversation":"conv_ticket_4821"}}Set it batch-wide in metadata on POST /v1/batches when every line
belongs to the same thread, or per line when they don't.
The identifier is yours to choose. Deriving it from something in your own schema (a ticket ID, a chat session ID) means you can jump from a support case straight to the model exchange that belongs to it.