If the charges stay outside the nucleus, the Fermi contact is zero. However, the Bohr-Weiskopf contribution can be nonzero, since the magnetic field can extend into the nucleus. Therefore, the spatial distribution of the magnetic moment inside the sphere matters for the energy correction, which would not be the case for a point nucleus.