Evaluating LLM Creativity through Long-Tail Performance
The paper argues that LLM creativity can be tested by how well models perform on rare input-output patterns.
The authors compare 15 benchmarks and 26 open-weight models, using n-gram absence from pretraining data as a proxy for novelty. They find creativity benchmarks sit farther into the long tail than standard benchmarks. Higher-tail benchmark slices also track more closely with human creativity judgments in LM-Arena. The paper proposes q@Tail(α) as a cheaper diagnostic for measuring task quality on the tail fraction of an existing benchmark. HF Daily Papers' note
The authors compare 15 benchmarks and 26 open-weight models, using n-gram absence from pretraining data as a proxy for novelty. They find creativity benchmarks sit farther into the long tail than standard benchmarks. Higher-tail benchmark slices also track more closely with human creativity judgments in LM-Arena. The paper proposes q@Tail(α) as a cheaper diagnostic for measuring task quality on the tail fraction of an existing benchmark. HF Daily Papers' note
score 5