Spectral differential imaging (SDI) is part of the observing strategy of current and on-going high-contrast imaging instruments on ground-based telescopes. Although it improves the star light rejection, SDI attenuates the signature of off-axis companions to the star, just like angular differential imaging (ADI). However, the attenuation due to SDI has the peculiarity of being dependent on the spectral properties of the companions. To date, no study has investigated these effects. Our team is addressing this problem based on data from a direct imaging survey of 16 stars combining the phase-mask coronagraph, the SDI and the ADI modes of VLT/NaCo. The objective of the survey is to search for cool (Teff<1000-1300 K) giant planets at separations of 5-10 AU orbiting young, nearby stars (<200 Myr, <25 pc). The data analysis did not yield any detections. As for the estimation of the sensivity limits of SDI-processed images, we show that it requires a different analysis than that used in ADI-based surveys. Based on a method using the flux predictions of evolutionary models and avoiding the estimation of contrast, we determine directly the mass sensivity limits of the survey for the ADI processing alone and with the combination of SDI and ADI. We show that SDI does not systematically improve the sensitivity due to the spectral properties and self-subtraction of point sources.