Semantic Caching
Semantic caching for LLM applications: reduce costs and latency by caching semantically similar queries with vector similarity.
Last updated
After this section you can
- Explain why exact-match caching misses most repeat intent in LLM apps, and walk the embed → vector search → threshold pipeline with the cost and latency of each path
- Calibrate a similarity threshold using hit rate versus false-hit risk, run the false-hit math, and defend a default in an interview
- Apply the production rules: 24h TTL, namespace-version invalidation on knowledge-base deploys, per-tenant isolation, and the never-cache list
- Compute the break-even hit rate and projected monthly savings from current per-token pricing
Semantic Caching
The cheapest lever in LLM system design: embed the question, search for a near-duplicate, and skip the model call entirely. Done right it cuts 20–40% of spend and 97% of latency on hit traffic; done wrong it serves wrong answers with total confidence.
Every LLM call you can avoid is money and latency you never spend — and on FAQ-shaped traffic, most questions have been asked before, just never in the same words. A semantic cache stores past questions as vectors and answers any new question that is close enough in meaning, turning a ~2-second, $0.003 model call into a ~50-millisecond, $0 lookup. This section covers the architecture, the threshold that makes or breaks it, the production rules, and the economics.
You have already seen every piece once: caches as a primitive in SD-1 · System Design 101, and a toy cache wired into the support bot in SD-3 · Your First Agentic System. What is new is the semantic part — matching on meaning instead of strings — and the judgment around thresholds, invalidation, and isolation.