Managing Context, Tokens, and Cost in Production OpenAI Applications

This is the third post in our OpenAI API fundamentals series, building on your first completion and function calling.

We noted in the first post in this series that the API has no server-side memory — every request sends the complete conversation from scratch. That fact has a direct, compounding cost consequence that's easy to miss until a real bill arrives: a 50-turn conversation's 51st message doesn't cost the same as its 1st. It costs roughly 50 times more, because the entire history rides along with every single request.

Tokens: the actual unit everything is billed in

Neither words nor characters map directly onto cost — the actual unit is a token, roughly ¾ of a word in English on average, though it varies by language and content.

import tiktoken
 
encoding = tiktoken.encoding_for_model("gpt-4o")
tokens = encoding.encode("Explain what a load balancer does.")
print(len(tokens))

7

Both input tokens (everything you send: system prompt, conversation history, the current message) and output tokens (the model's response) count toward cost — and pricing per token differs between the two, with output tokens typically priced higher. This is why an application that sends a large system prompt on every single request, or accumulates a long conversation history, sees cost scale with total conversation length, not with the length of just the newest message.

Counting tokens before you send them

Because tiktoken runs the same tokenizer the API itself uses, it's worth checking a request's actual token count before sending it — both to catch an unexpectedly large payload before paying for it, and to stay under a model's maximum context window:

def count_message_tokens(messages, model="gpt-4o"):
    encoding = tiktoken.encoding_for_model(model)
    total = 0
    for message in messages:
        total += 4  # per-message overhead (role, formatting tokens)
        for value in message.values():
            total += len(encoding.encode(str(value)))
    return total
 
tokens = count_message_tokens(messages)
if tokens > 100_000:
    messages = trim_history(messages)

The += 4 per-message overhead is a real, if small, detail worth knowing: the token count isn't simply the sum of each message's content — role labels and message-boundary formatting add a few tokens per message too, which becomes a genuinely non-trivial overhead across a conversation with many short messages.

Why context grows linearly, and why that's a real problem

messages = [{"role": "system", "content": "You are a helpful assistant."}]
 
def chat(user_message):
    messages.append({"role": "user", "content": user_message})
    response = client.chat.completions.create(model="gpt-4o", messages=messages)
    reply = response.choices[0].message.content
    messages.append({"role": "assistant", "content": reply})
    return reply

This is the naive, and extremely common, first implementation of a multi-turn chat: append every message, forever, and send the whole growing list every time. It works correctly. It also means turn 50 sends roughly 50 times the tokens of turn 1 — a cost curve growing without bound for as long as the conversation continues, entirely invisible until someone actually looks at token usage or the bill.

Fixing it: trimming, summarizing, or both

Sliding window — keep only the most recent N messages:

def trim_history(messages, keep_last=10):
    system = [m for m in messages if m["role"] == "system"]
    recent = [m for m in messages if m["role"] != "system"][-keep_last:]
    return system + recent

Simple, and appropriate when older conversation context genuinely stops being relevant after a while — a support chat about today's specific issue rarely needs message 1 by message 40.

Summarization — periodically compress older turns into a shorter summary, using the model itself:

def summarize(messages_to_compress):
    summary_prompt = "Summarize this conversation in 2-3 sentences, keeping key facts:"
    response = client.chat.completions.create(
        model="gpt-4o-mini",  # a smaller, cheaper model for a simple internal task
        messages=[
            {"role": "system", "content": summary_prompt},
            {"role": "user", "content": str(messages_to_compress)},
        ],
    )
    return response.choices[0].message.content

Preserves the gist of older context at a fraction of the token cost of the original messages, and — notice the model choice — this internal, low-stakes summarization step is exactly the right place to use a smaller, cheaper model rather than defaulting to your main, more expensive one for every single call in the system.

A production system often combines both: summarize older turns beyond a threshold into a compact block, keep a sliding window of the most recent few turns verbatim (since exact recent wording often matters more than older gist), and send the summary plus the recent window as the effective history — bounded total size, regardless of how long the actual conversation has run.

Matching model choice to task, deliberately

Not every call in a real application needs the most capable available model. A classification task ("is this support ticket urgent, yes or no") or the summarization step above genuinely doesn't need the same model handling nuanced customer-facing conversations. Routing internal, well-defined, lower-stakes tasks to a smaller model (gpt-4o-mini rather than gpt-4o, in OpenAI's current lineup) is one of the highest-leverage, lowest-effort cost optimizations available — often a 90%+ per-token cost reduction for tasks that don't need the larger model's additional capability at all.

Prompt caching: paying less for repeated context

For applications sending the same large system prompt or reference document on every request, OpenAI's prompt caching automatically discounts the portion of input tokens that exactly match a previous request within a short time window — a meaningful cost reduction for exactly the "large, mostly-static system prompt sent on every call" pattern that's extremely common in real applications, requiring no code change to benefit from beyond structuring the static portion of a prompt consistently.

Cache hits depend specifically on the prefix of a request matching a prior one exactly — this is why it matters to structure a prompt with static content (system instructions, reference documents) first, and dynamic content (the specific user's current message) last. A prompt that interleaves dynamic values throughout an otherwise-static system message breaks the exact-prefix match caching relies on, forfeiting a discount that a small reordering would have preserved.

Setting a hard budget, not just watching usage

Beyond the optimizations above, OpenAI's dashboard supports configuring hard usage limits at the organization or project level — a genuinely useful backstop against the "a bug in a retry loop calls the API a thousand times before anyone notices" failure mode, which token-level optimization alone doesn't protect against. Treating a spending cap as a deliberate safety control, the same way a cloud budget alert is treated for infrastructure spend, is worth setting up before a real production launch, not after a surprising invoice.

Batch API: a further discount for genuinely non-urgent work

For workloads that don't need a synchronous response — nightly classification of the day's support tickets, bulk summarization of a document set — OpenAI's Batch API processes requests asynchronously, within a 24-hour window, at a meaningfully discounted price relative to standard synchronous calls:

batch_input = [
    {"custom_id": f"ticket-{i}", "method": "POST", "url": "/v1/chat/completions",
     "body": {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": ticket_text}]}}
    for i, ticket_text in enumerate(todays_tickets)
]
 
with open("batch_input.jsonl", "w") as f:
    for item in batch_input:
        f.write(json.dumps(item) + "\n")
 
batch_file = client.files.create(file=open("batch_input.jsonl", "rb"), purpose="batch")
batch = client.batches.create(input_file_id=batch_file.id, endpoint="/v1/chat/completions", completion_window="24h")

This is the same "scale-to-zero, pay only for what you actually use, tolerate slower turnaround" trade-off covered for SageMaker's asynchronous inference and serverless endpoints later in this blog — appropriate specifically for genuinely non-urgent, high-volume workloads, and a poor fit for anything a user is actively waiting on.

Tracking cost per feature, not just total spend

A single aggregate OpenAI bill tells you total spend, but not which specific feature or call site is actually driving it. Tagging requests (via a custom header or simply logging alongside each call) with the originating feature is what makes per-feature cost attribution possible:

def logged_completion(feature_name: str, messages: list):
    response = client.chat.completions.create(model="gpt-4o", messages=messages)
    logger.info(f"feature={feature_name} tokens={response.usage.total_tokens} cost_estimate={estimate_cost(response.usage)}")
    return response

This is the same tagging discipline covered for Terraform resources and SageMaker training jobs elsewhere in this blog — without it, a genuinely expensive feature (an over-eager summarization step, an unnecessarily verbose system prompt) is invisible in an aggregate bill, indistinguishable from every other call in the system.

What to actually remember from this post

• Tokens, not words or characters, are the actual billing unit — and both input and output tokens count, typically at different rates.
• An unbounded conversation history means unbounded, linearly-growing per-turn cost — invisible until someone actually checks usage; count tokens before sending to catch it early.
• Sliding windows and periodic summarization are the two standard fixes, often combined — trim what's no longer relevant, or compress it, rather than sending the entire unbounded history forever.
• Route internal, low-stakes tasks (classification, summarization) to a smaller, cheaper model — one of the highest-leverage cost optimizations available, and one of the easiest to implement.
• Structure prompts with static content first, dynamic content last to actually benefit from prompt caching's exact-prefix matching.
• Set a hard organizational spending cap, not just token-level optimizations — a real backstop against a runaway bug, not just a usage-efficiency improvement.

Next in the series: Building a Retrieval-Augmented Generation (RAG) Pipeline with OpenAI, where we cover how to ground the model's answers in your own actual documents, rather than relying on what it happened to learn during training.