You are absolutely right. Whenever people live together in a community, there have to be rules, or maybe even morals, that everyone needs to obey. The Non Aggression Principal says that you can do as you please, as long as you don't hurt anyone.
But the personal freedom of individuals should never be sacrificed for the sake of the community. But this is what we witness right now.