Semantic Caching
When a cache may answer a rephrased question: the scope key that stops answers crossing users and tenants, the verifier that stops similar-but-wrong hits, freshness, and the break-even against the tier the hit replaces.
Last updated
After this section you can
- Compute the break-even hit rate and wrong-hit share for the tier a cache replaces
- Write a scope key that includes a permission hash, and state the store rule
- Choose a threshold and a verifier by the cost of a false hit, not the hit rate
07
Semantic Caching
A cache that answers rephrased questions: scope the key, verify the hit, price the miss.
Key idea
A semantic cache answers a new question with the stored answer to a similar one. A hit saves a model call; a wrong hit is a confident error no log records. Scope each entry, verify each hit, and price the break-even per tier.
Scope first, then the gate: a hit is served only when both pass
Amber: decisions and model calls. Teal: the scope key. Purple: memory. Pans sideways on a phone.