We constrain energy spectra of supernova neutrinos using the $\nu$-process light element synthesis in supernovae and the $^{11}$B abundance during Galactic chemical evolution. We calculate supernova nucleosynthesis due to the $\nu$-process assuming that neutrino energy spectra are Fermi-Dirac distributions with zero chemical potential. We investigate the dependence of the $^{11}$B yield on the total neutrino energy and the temperature of $\nu_{\mu,\tau}$ and $\bar{\nu}_{\mu,\tau}$. From the obtained yields and the contribution to the $^{11}$B yield from supernovae constrained by observed abundances and Galactic chemical evolution models, we find an acceptable range of the temperature of $\nu_{\mu,\tau}$ and $\bar{\nu}_{\mu,\tau}$ of 4.8 MeV to 6.6 MeV.