Techniques for training large neural networks
OpenAI documented infrastructure techniques for training large neural networks, including batch size scaling, distributed optimization, and cluster orchestration challenges.
Excerpt
Large neural networks are at the core of many recent advances in AI, but training them is a difficult engineering and research challenge which requires orchestrating a cluster of GPUs to perform a single synchronized calculation.
Read at source: https://openai.com/index/techniques-for-training-large-neural-networks