We measured the single-photon detection efficiency of NbN superconducting single-photon detectors as a function of the polarization state of the incident light for different wavelengths in the range from 488 nm to 1550 nm. The polarization contrast varies from ~5% at 488 nm to ~30% at 1550 nm, in good agreement with numerical calculations. We use an optical-impedance model to describe the absorption for polarization parallel to the wires of the detector. For the extremely lossy NbN material, the absorption can be kept constant by keeping the product of layer thickness and filling factor constant. As a consequence, the maximum possible absorption is independent of filling factor. By illuminating the detector through the substrate, an absorption efficiency of ~70% can be reached for a detector on Si or GaAs, without the need for an optical cavity.