Scaling Laws for Hypernetwork-Based Knowledge Injection in Large Language Models
The paper claims hypernetworks scale predictably as a way to inject factual knowledge into LLMs at train time.
The authors train a hypernetwork to generate a fixed LoRA adapter from a large corpus of facts, then insert that adapter into a target model for question answering. They introduce MegaWikiQA, with tens of millions of multi-hop QA examples across 39 Wikidata5M-derived domains. Their experiments report power-law scaling across hypernetwork depth, width, and target model size, plus stronger out-of-distribution scaling than LoRA finetuning or full finetuning in their evaluations. HF Daily Papers' note
The authors train a hypernetwork to generate a fixed LoRA adapter from a large corpus of facts, then insert that adapter into a target model for question answering. They introduce MegaWikiQA, with tens of millions of multi-hop QA examples across 39 Wikidata5M-derived domains. Their experiments report power-law scaling across hypernetwork depth, width, and target model size, plus stronger out-of-distribution scaling than LoRA finetuning or full finetuning in their evaluations. HF Daily Papers' note
score 4