During the 60's more people opposed the Apollo program spending than supported it, specifically for the reasons you allude to, spend that money on social programs, etc... De-funding these types of programs has been a major talking point of almost every election since then. The discussion has even been renewed quite a bit since SpaceX has came into existence.

https://www.theatlantic.com/technology/archive/2012/09/moond...