Coronal mass ejections observed in the corona exhibit a three-part structure, with a leading bright front indicating dense plasma, a low density cavity thought to be a signature of the embedded magnetic flux rope, and the high density core likely containing cold, prominence material. When observed in-situ, as Interplanetary CMEs (or ICMEs), the presence of all three of these signatures remains elusive, with the prominence material rarely observed. We report on a comprehensive and long-term search for prominence material inside ICMEs as observed by the Solar Wind Ion Composition Spectrometer on the Advanced Composition Explorer. Using a novel data analysis process, we are able to identify traces of low charge state plasma created during prominence eruptions associated with ICMEs. We find that the likelihood of occurrence of cold material in the heliosphere is vastly lower than that observed in the corona but that conditions during the eruption do allow low charge ions to make it into the solar wind, preserving their expansion history. We discuss the implications of these findings.