Most of neodymium magnets have a thin nickel-copper-nickel coating to protect them from erosion. This coating could splinter or crack due to collision or large pressure. This makes them vulnerable to environmental influences like moisture and they could oxidise.
Separate big magnets, especially spheres, with a piece of cardboard.
Avoid collisions of magnets as well as repeated mechanical exposure (e.g. blows, bashes).