Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection
The paper claims SSMs can turn retrieved documents into injectable hidden states, cutting query prefill from about 27 seconds to under 6 milliseconds on edge hardware.
PRECOG pre-encodes corpora offline as fixed-size SSM hidden states, then injects the best match at query time instead of re-reading retrieved context. The authors say answer quality matches in-context RAG on their TENNs-LLM test system, a 1.2B-parameter gated SSM with a 192 KB hidden state. They also describe SMC, a persistent memory scheme that consolidates episodic states into longer-term semantic memory. The paper argues this state-injection path is unavailable to Transformer KV-caches because they are position-entangled and grow with context length. ArXiv · AI/CL/LG's note
PRECOG pre-encodes corpora offline as fixed-size SSM hidden states, then injects the best match at query time instead of re-reading retrieved context. The authors say answer quality matches in-context RAG on their TENNs-LLM test system, a 1.2B-parameter gated SSM with a 192 KB hidden state. They also describe SMC, a persistent memory scheme that consolidates episodic states into longer-term semantic memory. The paper argues this state-injection path is unavailable to Transformer KV-caches because they are position-entangled and grow with context length. ArXiv · AI/CL/LG's note
score 5