What repeated LLM calls actually cost you

The bill arrives and it is bigger than last month. Everyone agrees usage is up, someone says the word "caching", and the conversation moves on because nobody knows how much of the spend is actually repeat work.

That number is worth having, and it is easier to get than it sounds. This post is about measuring your own repeat rate first, then deciding whether caching is worth doing at all.

Why guessing does not work here

Repeat rates vary enormously by workload, which is why quoting someone else's figure is useless.

A support assistant answering free-text questions from thousands of different users repeats almost nothing. A pipeline that classifies incoming documents against a fixed prompt, or enriches records where the same company name shows up repeatedly, or re-runs an evaluation set on every deploy, repeats a great deal. Same provider, same model, wildly different amounts of waste.

So the only number that matters is yours.

The arithmetic, once you have your repeat rate

The shape of it is simple. Take your monthly spend on the API, multiply by the fraction of calls that are exact repeats, and that is the ceiling of what caching can save you.

Work a hypothetical to see the shape. Say you make 10,000 calls a day and your average call costs one cent, so roughly $3,000 a month. Substitute your own figures here, because both of those numbers are yours rather than ours:

Repeat rate	Monthly saving at that spend
5%	$150
20%	$600
40%	$1,200

At 5% this is probably not worth your afternoon. At 40% it pays for itself the week you ship it. Which of those you are looking at is the only thing you need to know, and most teams have never measured it.

Two things the table understates. Cached responses do not consume rate limit or quota, which matters more than the money when you are close to a ceiling. And a cache hit returns in single-digit milliseconds instead of seconds, so the user-facing latency improvement is real even though it is not the reason to do this.

Measuring your repeat rate

You do not need to build the cache to find out. Hash the inputs, count how often each hash recurs, and read the counters back after a week.

import { app, Datastore } from 'codehooks-js';
import crypto from 'crypto';

app.post('/complete', async (req, res) => {
  const db = await Datastore.open();

  // Hash everything that should change the answer, not just the prompt
  const hash = crypto
    .createHash('sha256')
    .update(JSON.stringify({ model: req.body.model, prompt: req.body.prompt }))
    .digest('hex');

  // Count calls, and count how many were repeats
  await db.incr('llm:calls', 1);
  const seen = await db.get(`llm:seen:${hash}`);
  if (seen) {
    await db.incr('llm:repeats', 1);
  } else {
    await db.set(`llm:seen:${hash}`, '1', { ttl: 2592000000 }); // 30 days
  }

  // Still call the API. This measures only, it does not cache yet
  res.json(await callLlm(req.body.prompt));
});

app.get('/llm-stats', async (req, res) => {
  const db = await Datastore.open();
  const calls = Number(await db.get('llm:calls')) || 0;
  const repeats = Number(await db.get('llm:repeats')) || 0;

  res.json({
    calls,
    repeats,
    repeatRate: calls ? Math.round((repeats / calls) * 100) + '%' : 'no data'
  });
});

export default app.init();

Deploy that, leave it a week, and read /llm-stats. Now you have a real number, and the decision about whether to cache becomes arithmetic.

Note the TTL on the seen-markers. Thirty days keeps the measurement honest and stops the keyspace growing without limit.

Turning the measurement into a cache

If the rate justifies it, the change is small. You already have the hash, so store the answer next to the marker and return it on a hit.

app.post('/complete', async (req, res) => {
  const db = await Datastore.open();
  const key = 'llm:' + crypto
    .createHash('sha256')
    .update(JSON.stringify({ model: req.body.model, prompt: req.body.prompt }))
    .digest('hex');

  const hit = await db.get(key);
  if (hit) return res.json(JSON.parse(hit));

  const answer = await callLlm(req.body.prompt);
  await db.set(key, JSON.stringify(answer), { ttl: 86400000 }); // 1 day

  res.json(answer);
});

The lookup costs about 2ms. We measured that rather than estimating it: 1,500 sequential reads of roughly 1KB values, timed inside the handler around the read itself, holding under 3ms at the 95th percentile. Against a call that takes a second or more, the check is free.

What about the provider's own caching?

Worth addressing, because most providers now offer prompt caching and it is genuinely useful. It solves a different problem though, and the two work together rather than replacing each other.

Provider prompt caching caches a prefix of your prompt: the system prompt, a long document, a fixed set of instructions. On a repeat you pay a reduced rate for that prefix instead of full price.

What it does not do is skip the call. The request is still made, the model still generates, and output tokens are still billed in full. Entries also expire after a short window, so a prefix has to be reused fairly promptly to pay for itself, and some providers charge a premium to write the entry in the first place. The exact discounts, write costs and expiry times move, so check the source rather than an article: Anthropic, OpenAI, Google.

	Provider prompt caching	Response cache
What is cached	The prompt prefix	The whole response
Does the call happen?	Yes	No
Output tokens billed	Yes, full price	None
Time to answer	Full generation time	A few milliseconds
Helps when	Many different questions share one long context	The same request repeats

So they cover different waste. Prompt caching cuts the input cost of distinct calls that share a big prefix, which is the common case for RAG and long-document work. A response cache eliminates identical calls completely, output tokens included.

Most setups want both: prompt caching for the questions that differ, a response cache for the ones that don't. If your repeat rate came back high, the response cache is the bigger win, because it is the only one of the two that gets you to zero.

What is safe to cache, and what is not

This is where judgement is needed, and it is worth thinking about before you ship.

Usually safe. Classification and extraction against a fixed prompt. Embeddings, which are deterministic for the same input and the same model. Translation of identical strings. Anything where you would be annoyed to get a different answer for the same input.

Be careful. Anything where variation is the point, such as creative generation or deliberately sampling several answers. Anything personalised, where the same prompt from two users should not produce the same answer, unless you include the user in the key. Anything where the underlying data changes but the prompt does not, which is the case a TTL exists for.

Do not cache anything where a stale answer causes harm rather than annoyance. Pricing, availability, compliance decisions.

The key is where you encode this. Hash everything that should change the answer, including the model name and any parameters. If two calls should be allowed to differ, something in the key has to differ too.

How long should entries live?

Long enough to catch the repeats, short enough that a stale answer is not a problem. A day is a reasonable default for most content work.

Where fetching is expensive and reads are constant, a scheduled job can refresh entries before they expire, so nobody ever waits for a miss. Where the answer depends on data you control, purge the key when that data changes rather than waiting for the TTL.

Summary and conclusion

Start by measuring what fraction of your calls are exact repeats, which takes a hash, two counters and a week of waiting. A caching strategy chosen before that number exists is a guess.

Multiply that fraction by your bill. If the answer is small, you have saved yourself the work. If it is large, the cache itself is about fifteen lines, and the lookup adds a couple of milliseconds.

If you would rather not run a Redis instance to hold what is essentially a lookup table, caching metered API calls covers doing it in the same deploy as the handler.