Quantifying the failure of symmetry learning in linear diffusion models due to initialization and finite-sample effects