Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone
The paper proposes a scalar, NSC, meant to score architecture capacity from the network specification alone.
NSC is computed from the singular-value spectrum of weight matrices under random initialization, without data, gradients, or even instantiating the model. The authors say its additive layer structure enables an exact dynamic-programming search, NSC-DP, that finds the highest-NSC architecture under resource limits in seconds on CPU. They report stronger ranking performance than parameter count, FLOPs, and other training-free proxies across Transformer and CNN families. Their examples include a Transformer-XL variant on WikiText-103 and pruning LLaMA-7B to 5.7B without calibration data. HF Daily Papers' note
NSC is computed from the singular-value spectrum of weight matrices under random initialization, without data, gradients, or even instantiating the model. The authors say its additive layer structure enables an exact dynamic-programming search, NSC-DP, that finds the highest-NSC architecture under resource limits in seconds on CPU. They report stronger ranking performance than parameter count, FLOPs, and other training-free proxies across Transformer and CNN families. Their examples include a Transformer-XL variant on WikiText-103 and pruning LLaMA-7B to 5.7B without calibration data. HF Daily Papers' note
score 4