At least devices like the Alexa, Google Home, etc are developed by large companies; the "invasion of privacy" that happens is at least performed by a relatively-reputable actors that have decent internal security so you at least know WHO is invading your privacy.
The real fear should be with less intrusive "smart home" devices created by smaller companies. People are more likely to trust a single smart home "security" camera than they are to trust an assistant like Alexa or Google Home. Many of these devices have horrible security and wind up entangled in untargeted botnet clusters anyways; when a targeted attack happens on these devices, your privacy can then become violated by UNKNOWN actors.
Smart home devices are disastrous for privacy, and it isn't just the full-scale assistants developed by large companies.