Rethinking Neural Thickets Without Weight Perturbations
Random output biases matched or beat full-weight perturbations on most reasoning tasks, with the model entirely frozen.
The authors reproduced the reported Neural Thickets gains but found they did not require an explanation based on nearby experts in weight space. Their Perturb-Select-Vote procedure generates varied predictors, selects strong performers on a small validation set, and combines their predictions by voting. A first-order analysis suggests small weight perturbations work primarily through shifts in output logits. In vision tests under domain shift, including Earth observation, logit-based PSV substantially improved performance, with little additional benefit from perturbing all weights.
ArXiv · AI/CL/LG's note
The authors reproduced the reported Neural Thickets gains but found they did not require an explanation based on nearby experts in weight space. Their Perturb-Select-Vote procedure generates varied predictors, selects strong performers on a small validation set, and combines their predictions by voting. A first-order analysis suggests small weight perturbations work primarily through shifts in output logits. In vision tests under domain shift, including Earth observation, logit-based PSV substantially improved performance, with little additional benefit from perturbing all weights.
ArXiv · AI/CL/LG's note
score 5