Massive star clusters in the Magellanic Clouds are observed to follow a striking trend in size with age – older clusters exhibit a much greater spread in core radius than do younger clusters, which are generally compact. We present results from realistic N-body modelling of massive star clusters, aimed at investigating a dynamical origin for the radius-age trend. We find that stellar-mass black holes, formed as remnants of the most massive stars in a cluster, can constitute a dynamically important population. If retained, these objects rapidly form a dense core where interactions are common, resulting in the scattering of black holes into the cluster halo, and the ejection of black holes from the cluster. These processes heat the stellar component, resulting in prolonged core expansion of a magnitude matching the observations. Core expansion at early times does not result from the action of black holes, but can be reproduced by the effects of rapid mass-loss due to stellar evolution in a primordially mass segregated cluster.