A Three-Layer Caching Architecture for Low-Latency LLM Web Search on Commodity CPU Hardware
OreoLook reports an 89.3% Redis hit rate while running its search stack on a single commodity CPU server.
The paper describes a three-layer cache for an open-source LLM answer engine: session context in Redis with compressed disk overflow, semantic query matching for rephrased searches, and URL embedding reuse across sessions. Answer synthesis still runs through a remote inference provider. The evaluated setup used one 8-vCPU Intel Cascade Lake machine with 32 GB RAM, 30 Hypercorn workers, and three containerized replicas. The authors report 0.1 ms Redis read latency and 1.38 MB of memory overhead. HF Daily Papers' note
The paper describes a three-layer cache for an open-source LLM answer engine: session context in Redis with compressed disk overflow, semantic query matching for rephrased searches, and URL embedding reuse across sessions. Answer synthesis still runs through a remote inference provider. The evaluated setup used one 8-vCPU Intel Cascade Lake machine with 32 GB RAM, 30 Hypercorn workers, and three containerized replicas. The authors report 0.1 ms Redis read latency and 1.38 MB of memory overhead. HF Daily Papers' note
score 4