Long conversations

A chat request carries the whole conversation every time. Three settings keep a long one fast and cheap. Each is opt-in, and each changes only what you ask it to.

Send a file once

Upload a picture, recording, video, PDF or text file with your API key:

curl https://api.katara.com/v1/files \
  -H "Authorization: Bearer $KATARA_API_KEY" \
  -F file=@contract.pdf

{ "id": "file_3f9a1c0b7d2e5a6b8c9d0e1f", "object": "file", "bytes": 482113, "filename": "contract.pdf", "mime_type": "application/pdf" }

Then name it in a message, on this turn and every later one:

{ "role": "user", "content": [
  { "type": "text", "text": "Which clause covers termination?" },
  { "type": "file", "file": { "file_id": "file_3f9a1c0b7d2e5a6b8c9d0e1f" } }
] }

The model gets the file in the form it reads, with a short line before it that names the file id. A file can be up to 20 MB, and the media in one request still add up to at most 20 MB. When you also send extracted_text with the upload, a model that cannot read that kind of file gets the text instead. Only your account can use, read (GET /v1/files/{id}) or delete (DELETE /v1/files/{id}) a file. Files need an API key: without one, these routes answer 401 invalid_api_key, and a paid request without a key that names a file is refused with 400 invalid_request. The file’s content is billed as input tokens each time it is sent to a model, as an inline file is.

Summarise old turns

"katara": { "compaction": { "trigger_tokens": 150000, "keep_turns": 2 } }

When the request’s input passes trigger_tokens, Katara has a fast model write a summary of every turn before the last keep_turns. The summary keeps the files’ ids and names, what was used from them, and names, numbers and links word for word. The model answers from the summary and the recent turns, and the reply returns the summary:

"katara": {
  "compaction": {
    "message": { "role": "system", "content": "Summary of the earlier conversation:\n\n## Topic\n…" },
    "replaces_from": 1,
    "replaces_to": 14,
    "summary": { "cost_usdc": "0.004210", "usage": { "prompt_tokens": 151204, "completion_tokens": 1830 } }
  }
}

On your next request, send message in place of messages[replaces_from:replaces_to] and keep the rest. Until the conversation passes the trigger again nothing more is summarised, and the start of your requests stays the same. When it passes it again, the earlier summary is folded into the new one. The summary is billed to you as a request of its own, at that model’s price; the reply’s summary field shows its cost and usage, and the reply’s own cost does not include it. If the summary cannot be made, the request goes whole and compaction.skipped says so. Requests without an API key cannot ask for compaction.

Clear old tool results

"katara": { "context_edits": { "keep_turns": 3, "trigger_tokens": 10000 } }

Once the tool results in a request add up to trigger_tokens, results from older turns are replaced with a one-line note naming the tool, so the model knows to call it again if it needs the result. The reply’s katara.context_edits says how many were cleared.

What is cleared grows in steps of keep_turns turns: with 3, the results of turns 1 to 3 are cleared from the sixth turn, turns 1 to 6 from the ninth, and so on. Between steps no earlier message changes, so the conversation’s start keeps being read from cache; clearing one more turn on every request would change an earlier message every time. The cost is that a result can stay for up to five turns before it goes.

Keep the start the same

Katara’s own models read the part of a prompt that matches an earlier one from cache, at a tenth or less of the input price (Caching). Everything above helps it: a file id is the same on every turn, a summary stays put until the next one, and tool results are cleared in steps. For the rest, send the same system prompt and tools in the same order, and add new turns at the end.