Skip to main content

API cache and proxy

Stop paying twice for the same API call

A cache in front of any metered API, with no Redis to run, no instance to size and no second service to keep alive.

Start free →See the cache pattern →

Key-value store included · Nothing to size · Data stored in the EU

The bill is the problem, not the milliseconds

Every repeated call to a metered API is money you have already spent, spent again. With LLM providers this adds up quickly, because the same prompt, the same document and the same enrichment lookup get charged every time they go out.

The standard fix is Redis. That means another managed service, another instance to size, another bill and another dependency to keep alive, all for what is essentially a lookup table sitting in front of a request.

Codehooks puts the key-value store and the request handler in the same deploy, so caching, rate limiting and key rotation live where the traffic already passes.

What the usual fix actually costs you

Redis or Upstash, plus your proxyCodehooks
A cache service, billed separatelyKey-value store included
Somewhere to run the proxy itselfHandler and store in one deploy
Instance sizing and eviction tuningNothing to size
API keys held in a third placeKeys and rotation beside the handler
Two dashboards and two failure modesOne deploy

Cache-aside in front of a metered API

The whole thing. Notice that no second service appears anywhere in the file: no cache to connect to, no instance to size, no separate set of credentials.

index.js
import { app, Datastore } from 'codehooks-js';
import crypto from 'crypto';

app.post('/complete', async (req, res) => {
  const db = await Datastore.open();
  // Hash everything that should change the answer, not just the prompt
  const key = 'llm:' + crypto
    .createHash('sha256')
    .update(JSON.stringify({
      model: req.body.model,
      prompt: req.body.prompt
    }))
    .digest('hex');

  // Already paid for this one
  const hit = await db.get(key);
  if (hit) return res.json(JSON.parse(hit));

  const answer = await callLlm(req.body.prompt);
  await db.set(key, JSON.stringify(answer), { ttl: 86400000 });

  res.json(answer);
});

export default app.init();

The cache policy is code, so which keys are reusable and what TTL each route deserves can be tuned per route rather than in a dashboard. That part is worth handing to a coding agent.

Questions worth asking first

Including the one where a dedicated cache service is the better answer

Why not just use Upstash? It is cheap.
Cost is not the argument, and we should not pretend it is. Upstash is inexpensive and good at what it does. The argument is that the proxy has to run somewhere regardless, and here the store comes with it. You are removing a service from the path rather than saving money on one: one deploy, one set of credentials, one thing to keep alive instead of two.
Is it fast enough to sit in the request path?
A cache hit adds about 2ms, and stays under 3ms at the 95th percentile. That is measured rather than estimated: 1,500 sequential reads of roughly 1KB values, timed inside the handler around the read itself. The first request after a deploy is slower, around 5ms. For context, the calls you are caching usually take hundreds of milliseconds to several seconds, so the lookup is not the part that matters. One caveat worth knowing: those reads are sequential, so a handler doing a few lookups is fine, while one looping over hundreds of keys wants getMany or parallel reads instead.
My provider already offers prompt caching. Isn't that the same thing?
No, and the two work together. Provider prompt caching caches a prefix of your prompt, so a long system prompt or document costs less on a repeat. What it does not do is skip the call: the request still happens, the model still generates, and output tokens are still billed in full. A response cache skips the call entirely, which takes the cost to zero rather than reducing it. Use prompt caching for the many questions that share one long context, and a response cache for the requests that are genuinely identical.
What about cache invalidation?
Three options, and most setups use the first. A TTL on write expires entries without you doing anything ({ ttl: 86400000 } for a day). An explicit purge removes a key when you know the underlying data changed. And a scheduled job can refresh entries before they expire, which suits data that is expensive to fetch and read constantly.
Can I cache things other than LLM responses?
Yes, anything you pay per call for. Enrichment lookups, geocoding, currency rates, third-party product data. The pattern is the same: hash the inputs into a key, check before calling, store the result with a TTL that matches how stale the answer is allowed to be.
Where is the data stored?
In the EU, on servers in Amsterdam and Ireland.

Stop paying for the same answer twice

A key-value store and a request handler in one deploy, so the cache lives where the traffic already passes. The rest of the platform is on the Codehooks homepage.

Start freeRead the docs

Nothing to size · Data stored in the EU