Decades of direct, indirect and collider searches of dark matter have set stringent constraints pushing us to entertain the possibility that dark matter interacts with us purely gravitationally. In this nightmare scenario, the underlying particle physics of dark matter as well as our cosmological history could still be discerned purely gravitationally by studying the gravitational substructure that dark matter collapses into. In this talk, I present stringent limits on this substructure arising from the non-observation of heat exchange between substructure contained in ultra-faint dwarfs, and the stars, which would cause the expansion of the stellar half-light radius. These limits also translate to orders-of-magnitude improvement in limits on the primordial power spectrum down to the kpc scale.