How to Cut LLM API Costs in Production (With Go Code)

By Hamza Anis, CEO, CapregSoft

The fastest way to cut LLM API costs in production is to stop paying for the same tokens twice. Put the stable part of every prompt first and turn on provider prompt caching. Answer repeated questions from an exact or semantic cache before calling a model. Route easy requests to smaller models. Cap output length. Do it in that order, and measure tokens per feature before and after each step so you know which change actually moved the bill.

That's the whole strategy. The rest of this post is the engineering: where the money actually goes, how each layer works, what it can't do, and a small Go gateway that implements all of it with a metric on every hop.

Where LLM spend actually goes

Before optimizing anything, split your bill into four buckets. In most production systems we look at, the surprise is how little of the spend is the "real" work:

• Repeated context.

  System prompts, tool schemas, few-shot examples, and retrieved documents resent on every call. This is usually the largest input line.

• Output tokens.

  Output is priced higher than input at every major provider, and verbose answers, chain-of-thought left on in production, and "explain your reasoning" prompts add up fast.

• Retries and fallbacks.

  Timeouts that retry the full request, JSON-parse failures that re-ask the model, and agent loops that call the model five times to do one thing.

• Duplicate questions.

  The same (or nearly the same) question asked by different users: support deflection, internal knowledge bots, classification of similar tickets.

You can't reduce what you can't attribute, so the first job is accounting: tag every call with the feature that made it, the model, input tokens, cached input tokens, output tokens, and whether it was a retry. The OpenTelemetry GenAI semantic conventions give you standard attribute names for this, so the data lands in whatever observability stack you already run.

A week of that data answers the questions that matter: which feature costs the most per request, how much of the input is repeated prefix, and how many requests are near-duplicates.

Layer 1: Prompt caching (do this first)

Anthropic, OpenAI, and Google all cache repeated prompt prefixes and bill cached input tokens at a steep discount. Anthropic charges cache reads at a small fraction of the base input price after a one-off cache write; OpenAI applies caching automatically to long prompts. Check the current Anthropic and OpenAI docs for exact rates, minimum prefix lengths, and TTLs, because they change.

The engineering rule is simple: the cache matches on the prefix, so everything stable goes first, and everything that varies goes last.

• Order: system instructions → tool definitions → few-shot examples → long reference documents → conversation history → the new user message.

• Never put a timestamp, request ID, or user name near the top of the prompt. One changed byte early in the prompt invalidates everything after it.

• Keep tool schemas byte-identical between calls. Generating them from a map with random iteration order (a classic Go mistake) silently breaks caching. Sort keys.

Two honest caveats that most "cut costs 90%" posts skip:

1. Prompt caching only discounts

   input

   tokens. It does nothing for output tokens, so a chatty model stays expensive.

2. It only helps when the prefix is actually repeated within the cache TTL. Low-traffic features with long gaps between calls may see little benefit.

Layer 2: Exact-match caching

If the full request is identical (same model, same prompt, same parameters) and the answer doesn't depend on who's asking or when, don't call the model at all. Hash the canonicalized request and look it up in Redis.

This catches more than you'd expect: classification of templated inputs, extraction on re-uploaded documents, and deterministic transformation jobs that get re-run after failures. It's also zero-risk to correctness, because it only ever returns an answer to the exact same question.

Layer 3: Semantic caching

Semantic caching goes further: embed the incoming question, look for a previously answered question within a similarity threshold, and return that answer. A hit skips the model call entirely, so unlike prompt caching it saves output tokens too.

It is also the layer that can hurt you, so treat it like a feature with its own acceptance criteria:

• Namespace everything.

  Cache per feature, and per tenant when answers depend on tenant data. A support answer for Customer A must never be served to Customer B.

• Only cache "public" answers.

  If the right answer depends on the user's permissions, account state, or today's date, it is not cacheable.

• Start strict.

  Set a high similarity threshold, log every hit with both questions side by side, and review a sample before loosening it.

• Expire entries.

  Give entries a TTL that matches how often the underlying knowledge changes, and purge the namespace when you reindex documents.

Want a second pair of eyes on your LLM bill? We build and tune production AI systems, including gateways, caching, and routing. Book a technical discovery call and we'll tell you where your tokens are going, and whether it's worth fixing.

Layer 4: Model routing

Most production traffic doesn't need your most capable model. Classification, extraction, short rewrites, and routing decisions usually run fine on a small, fast model. Reserve the large model for multi-step reasoning and long-context synthesis.

Start with explicit rules, not a learned router:

• Route by

  task type

  , which the calling feature already knows ("classify", "extract", "draft", "reason").

• Route by

  input size

  : very long contexts go to models priced for it.

• Escalate on failure:

  if the small model's output fails validation (schema, confidence, a policy check), retry once on the larger model. You pay for the big model only on hard cases.

Keep an evaluation set for each routed task and re-run it whenever you change a route or a provider changes a model. A cheaper route that quietly lowers quality isn't a saving.

Layer 5: Output-token discipline

Because output tokens cost more and are never discounted by prompt caching, output length is often the cheapest lever left:

• Set

  max_tokens

  per feature, not one global ceiling.

• Ask for structured output (JSON with a schema) instead of prose when a program consumes the result.

• Turn off verbose reasoning in production paths that don't need it, and never ask the model to explain itself unless a human reads the explanation.

• Stream responses and stop reading when you have what you need.

A small LLM gateway in Go

You don't need a platform to do all of this. A thin gateway in front of your providers is enough to apply the layers in order and measure each one. Here is the core of one:

package gateway

import (
    "context"
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "time"

    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/metric"
)

type Message struct {
    Role    string `json:"role"`
    Content string `json:"content"`
}

type Request struct {
    Feature   string    `json:"-"` // who is paying: "support-bot", "ticket-classifier", ...
    Namespace string    `json:"-"` // semantic-cache scope, e.g. feature + tenant
    Task      string    `json:"-"` // "classify", "extract", "draft", "reason"
    Cacheable bool      `json:"-"` // false if the answer depends on user, permissions or time
    Model     string    `json:"model"`
    System    string    `json:"system"` // stable prefix first: keeps provider prompt caching effective
    Messages  []Message `json:"messages"`
    MaxTokens int       `json:"max_tokens"`
}

type Response struct {
    Text         string `json:"text"`
    InputTokens  int    `json:"input_tokens"`
    CachedTokens int    `json:"cached_tokens"`
    OutputTokens int    `json:"output_tokens"`
}

type Provider interface {
    Complete(ctx context.Context, req Request) (Response, error)
}

type ExactCache interface {
    Get(ctx context.Context, key string) (Response, bool)
    Set(ctx context.Context, key string, resp Response, ttl time.Duration)
}

type SemanticCache interface {
    Lookup(ctx context.Context, namespace string, vec []float32, minSimilarity float32) (Response, bool)
    Store(ctx context.Context, namespace string, vec []float32, resp Response, ttl time.Duration)
}

type Embedder interface {
    Embed(ctx context.Context, text string) ([]float32, error)
}

type Gateway struct {
    Provider      Provider
    Exact         ExactCache
    Semantic      SemanticCache
    Embed         Embedder
    Route         func(Request) string // returns the model to use
    MinSimilarity float32              // start strict, e.g. 0.95, and tune from logged hits
    TTL           time.Duration

    Requests metric.Int64Counter // attribute "layer": exact | semantic | model
    Tokens   metric.Int64Counter // attribute "kind": input | cached | output
}

func (g *Gateway) Complete(ctx context.Context, req Request) (Response, error) {
    feature := attribute.String("feature", req.Feature)

    // Key the cache on the request as the caller sent it, before routing
    // picks a model, so reads and writes always use the same key.
    var key string
    var vec []float32
    if req.Cacheable {
        key = cacheKey(req)
        if resp, ok := g.Exact.Get(ctx, key); ok {
            g.Requests.Add(ctx, 1, metric.WithAttributes(feature, attribute.String("layer", "exact")))
            return resp, nil
        }
        // Semantic lookup on the latest user turn: intended for single-turn
        // features (FAQ answers, classification), not open conversations.
        if v, err := g.Embed.Embed(ctx, lastUserMessage(req)); err == nil {
            vec = v
            if resp, ok := g.Semantic.Lookup(ctx, req.Namespace, vec, g.MinSimilarity); ok {
                g.Requests.Add(ctx, 1, metric.WithAttributes(feature, attribute.String("layer", "semantic")))
                return resp, nil
            }
        }
    }

    req.Model = g.Route(req)
    resp, err := g.Provider.Complete(ctx, req)
    if err != nil {
        return Response{}, err
    }

    g.Requests.Add(ctx, 1, metric.WithAttributes(feature, attribute.String("layer", "model"), attribute.String("model", req.Model)))
    g.Tokens.Add(ctx, int64(resp.InputTokens-resp.CachedTokens), metric.WithAttributes(feature, attribute.String("kind", "input")))
    g.Tokens.Add(ctx, int64(resp.CachedTokens), metric.WithAttributes(feature, attribute.String("kind", "cached")))
    g.Tokens.Add(ctx, int64(resp.OutputTokens), metric.WithAttributes(feature, attribute.String("kind", "output")))

    if req.Cacheable {
        g.Exact.Set(ctx, key, resp, g.TTL)
        if vec != nil {
            g.Semantic.Store(ctx, req.Namespace, vec, resp, g.TTL)
        }
    }
    return resp, nil
}

// cacheKey hashes the canonical request. json.Marshal emits struct fields in
// declaration order, so identical requests always produce identical keys.
func cacheKey(req Request) string {
    b, _ := json.Marshal(req)
    sum := sha256.Sum256(b)
    return hex.EncodeToString(sum[:])
}

func lastUserMessage(req Request) string {
    for i := len(req.Messages) - 1; i >= 0; i-- {
        if req.Messages[i].Role == "user" {
            return req.Messages[i].Content
        }
    }
    return ""
}

A few design choices worth copying:

• Cacheable

  is set by the caller, not guessed by the gateway.

  The feature that builds the prompt is the only code that knows whether the answer depends on the user.

• Namespace

  scopes semantic hits.

  Build it from the feature name plus the tenant ID for anything tenant-specific.

• Every exit path is counted.

  Hit rate per layer and tokens per feature fall straight out of the two counters, which is what you need to prove the savings to finance.

• Routing is a plain function.

  Start with a

  switch

  on

  req.Task

  ; you can replace it later without touching callers.

In production you'd add request timeouts, a circuit breaker per provider, and a retry policy that retries transport errors but never retries a request that already produced output tokens.

The rollout order that works

1. Instrument

   tokens per feature (1 week of baseline data).

2. Reorder prompts

   for prefix caching and enable it. Lowest risk, often the biggest single win.

3. Cap output

   per feature and switch machine-consumed outputs to structured JSON.

4. Add exact-match caching

   for deterministic, non-personal requests.

5. Route

   by task type, with an eval set per route and escalation on validation failure.

6. Add semantic caching

   last, per namespace, starting strict and reviewing hits.

Each step is independently measurable, and you can stop when the remaining spend is worth what it buys. The same thinking applies beyond the LLM line item: our Golang backend work is mostly about removing waste the architecture forces you to pay for.

If you are wiring LLM calls into multi-step workflows, read why AI agents fail in production, because retries and agent loops are where token spend quietly multiplies. If most of your prompt is retrieved context, the production RAG architecture guide shows how to retrieve less and better. And if agents reach your product through tools, building an MCP server in Go covers returning compact, cache-friendly tool results.

FAQ

What is the fastest way to reduce LLM costs?

Prompt caching. Put the stable part of every prompt (instructions, tool schemas, examples) first and enable the provider's cache; repeated prefixes are then billed at a fraction of the normal input price. It needs no new infrastructure and doesn't change answers.

Does semantic caching reduce output token costs?

Yes, but only on hits: a semantic cache hit skips the model call entirely, so you pay neither input nor output tokens. Prompt caching is different: it only discounts input tokens and never touches output cost.

Is semantic caching safe for every request?

No. Only cache requests whose correct answer doesn't depend on the user, their permissions, or the current time. Scope the cache by namespace (feature plus tenant where needed), set a strict similarity threshold, and give entries a TTL.

Should we build an LLM gateway or buy one?

Buy if you only need routing and caching across providers. Build a thin one when you need per-feature cost accounting, tenant-aware caching, or custom routing rules tied to your own data. It is a few hundred lines of Go, not a platform.

How much can we realistically save?

It depends on how repetitive your traffic is and how bloated your prompts are, so measure before promising a number. Instrument tokens per feature for a week, then apply the layers in order and track hit rate and cost per request for each. For illustration only (a hypothetical, not a client result): say a team spends $20,000/month, $12,000 on input tokens and $8,000 on output. If 60% of input tokens are a repeated prefix and cached reads bill at 10% of the normal input price, input drops to $12,000 × (0.4 + 0.6 × 0.1) = $5,520, so the bill is $13,520. Routing half of all requests to a model priced at one-fifth of the main one takes that to $13,520 × (0.5 + 0.5 × 0.2) = $8,112, of which $4,800 is output. Capping output length to cut output tokens by 25% saves another $1,200, leaving about $6,900/month, roughly a third of where you started. The assumptions do all the work here (cache-write charges, escalations back to the big model, and eval runs are ignored), so run the same arithmetic on your own week of data before quoting anyone a number.

Get your LLM spend under control

We're a senior backend and AI engineering team: we build AI automation that runs in production, not in a demo, with the cost and reliability controls above built in. Book a technical discovery call: 30 minutes with a senior engineer, and a straight answer on where your tokens are going and what it would take to fix it.