A lot of the technology we take for granted--camera phones, LEDs, CAT scans, wireless headsets--were developed in part or entirely by NASA to facilitate space travel before becoming mass produced. Even if going to space doesn't help us directly, the advancements we've made in the process have changed society for the better.
Plus, it's not like we aren't spending billions on worse things already.