Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views
Auxiliary reformulations improved pre-training learning even when they replaced repeated documents under the same token budget.
The paper uses controlled experiments to test whether LLMs acquire knowledge better from alternate views of the same knowledge. It finds repetition is needed, but paraphrasing only helps at smaller batch sizes. The authors report that auxiliary views help even on factual recall and do not depend on a stronger teacher model generating them. They also point to contextual and foundational knowledge as useful when prior knowledge is missing, with effects visible in layer-wise biases and compression. ArXiv · AI/CL/LG's note
The paper uses controlled experiments to test whether LLMs acquire knowledge better from alternate views of the same knowledge. It finds repetition is needed, but paraphrasing only helps at smaller batch sizes. The authors report that auxiliary views help even on factual recall and do not depend on a stronger teacher model generating them. They also point to contextual and foundational knowledge as useful when prior knowledge is missing, with effects visible in layer-wise biases and compression. ArXiv · AI/CL/LG's note
score 5