I am sorry if I was not clear, but these ones are not invasive, as I mentioned in the post. They are a sterile variety - the seeds cannot reproduce.
Check this out: https://www.fast-growing-trees.com/blog/royal-empress-trees-invasive-heres/
But you're right - humans are perhaps more invasive than anything. Still though there is a good point in not introducing a species into an environment where everything else won't know how to respond to it.
We also have Chinese Privet, which is considered invasive - it was growing here when we bought the land - but goats LOVE it. So, in that way, invasive plants can be useful - because we can rotate goats around to different stands of this stuff and they will eat it down and in a month it will grow back up.