Going Deeper With Memories
DeepMem reports better byte-level language modeling than a matched Transformer while preserving exact byte identity through compression.
The architecture compresses byte sequences into shorter representations, retaining their exact details in a lossless carrier and separate sidebands. Memory blocks and repeated computation work on the compressed representations before an attention-based decoder restores byte resolution. The paper reports stronger results on FineWeb-Edu and PG19, the best macro-average performance across the evaluated FLORES languages, and improved PIQA accuracy. Its bottleneck refinement uses approximately 7,600 times fewer counted FLOPs than equivalent full-resolution reasoning; that figure applies to refinement, not the entire model.
ArXiv · AI/CL/LG's note
The architecture compresses byte sequences into shorter representations, retaining their exact details in a lossless carrier and separate sidebands. Memory blocks and repeated computation work on the compressed representations before an attention-based decoder restores byte resolution. The paper reports stronger results on FineWeb-Edu and PG19, the best macro-average performance across the evaluated FLORES languages, and improved PIQA accuracy. Its bottleneck refinement uses approximately 7,600 times fewer counted FLOPs than equivalent full-resolution reasoning; that figure applies to refinement, not the entire model.
ArXiv · AI/CL/LG's note
score 5