A Dominant Diffuse Phase in the Sparse Autoencoder Phase Diagram
The SAE runs did not recover or merge the synthetic features; they settled into a dense, diffuse solution instead.
Across 200 initialized fits in ten grid cells, the paper reports zero full-dictionary recoveries and zero merges. Reconstruction was nearly perfect, but learned atoms stayed well below the 0.95 recovery criterion and codes were much denser than the ground truth. The same behavior held in robustness checks and 3,300 additional Adam fits across the full grid. ArXiv · AI/CL/LG's note
Across 200 initialized fits in ten grid cells, the paper reports zero full-dictionary recoveries and zero merges. Reconstruction was nearly perfect, but learned atoms stayed well below the 0.95 recovery criterion and codes were much denser than the ground truth. The same behavior held in robustness checks and 3,300 additional Adam fits across the full grid. ArXiv · AI/CL/LG's note
score 5