The short-wave stability properties of a Batchelor vortex are used to explain the intrinsic resistance of vortices to turbulent diffusion. We show that turbulence produced within the vortex core has to overcome a stabilizing ‘dispersion buffer’, where energy of the perturbations is dispersed by inertial waves without interfering with the mean flow, before they can reach the periphery of the vortex. While angular momentum is maintained by this mechanism, the difference in energy extraction by turbulence from the axial and tangential velocity fields due to a lack of alignment between the mean and turbulent strain tensors, a typical effect of flow rotation or curvature, leads to stabilization through a progressive damping of the axial shear in the vortex core. We show that the efficiency of these stabilizing mechanisms depends on the swirl number, the ratio between the maximum tangential velocity and the axial velocity difference. If the swirl parameter is low enough, turbulence is able to reach the vortex periphery and a small circulation overshoot develops, leading to weak diffusion of angular momentum outward.