Derivative Computation in PINNs: Automatic Differentiation, Finite Differences and Beyond
Finite differences matched autograd on the tested PINN problems while using less GPU memory.
The paper compares finite-difference derivative computation with automatic differentiation across three benchmark PDEs. With calibrated step sizes, finite differences matched AD accuracy, ran faster across the tested batch sizes, and used substantially less GPU memory. The authors also report a stochastic FD variant that beat AD on a stationary problem. For architectures with inter-sample dependencies such as BatchNorm or self-attention, they say the standard PyTorch autograd pattern can give silently incorrect per-sample derivatives, while the correct autograd alternative becomes infeasible at relevant batch sizes. ArXiv · AI/CL/LG's note
The paper compares finite-difference derivative computation with automatic differentiation across three benchmark PDEs. With calibrated step sizes, finite differences matched AD accuracy, ran faster across the tested batch sizes, and used substantially less GPU memory. The authors also report a stochastic FD variant that beat AD on a stationary problem. For architectures with inter-sample dependencies such as BatchNorm or self-attention, they say the standard PyTorch autograd pattern can give silently incorrect per-sample derivatives, while the correct autograd alternative becomes infeasible at relevant batch sizes. ArXiv · AI/CL/LG's note
score 5