Going Deeper With Memories
DeepMem compresses byte sequences for computation while preserving exact byte identity.
The architecture combines a learned invertible transform with a lossless carrier and detail sidebands, then restores byte resolution through an attention-based decoder. It outperforms a matched byte-level Transformer on FineWeb-Edu and PG19 and records the best macro-average performance across the evaluated FLORES languages. It also improves PIQA accuracy while remaining competitive on the other reported evaluations. The paper reports roughly 7,600 times fewer counted FLOPs for bottleneck refinement than equivalent full-resolution reasoning, a comparison specific to that computation.
HF Daily Papers' note
The architecture combines a learned invertible transform with a lossless carrier and detail sidebands, then restores byte resolution through an attention-based decoder. It outperforms a matched byte-level Transformer on FineWeb-Edu and PG19 and records the best macro-average performance across the evaluated FLORES languages. It also improves PIQA accuracy while remaining competitive on the other reported evaluations. The paper reports roughly 7,600 times fewer counted FLOPs for bottleneck refinement than equivalent full-resolution reasoning, a comparison specific to that computation.
HF Daily Papers' note
score 5