Large-scale Neuron Community Discovery Reveals Neuron-Sparse Behaviors in Language Models
Groups of at most 128 neurons explained at least one layer’s contribution on 10.3% of held-out prompts in Llama-3.1-8B.
The researchers analyzed ten million prompts and found thousands of candidate neuron groups that recur across inputs. Causal tests identified groups whose removal disrupted predictions and whose retention preserved predictions despite removing the other MLP neurons in the tested layers. Their functions included grammatical agreement, copying, addition and multiplication. Coverage fell to 8.7% when requiring explanations spanning at least four layers, and 3.3% for at least twelve.
ArXiv · AI/CL/LG's note
The researchers analyzed ten million prompts and found thousands of candidate neuron groups that recur across inputs. Causal tests identified groups whose removal disrupted predictions and whose retention preserved predictions despite removing the other MLP neurons in the tested layers. Their functions included grammatical agreement, copying, addition and multiplication. Coverage fell to 8.7% when requiring explanations spanning at least four layers, and 3.3% for at least twelve.
ArXiv · AI/CL/LG's note
score 5