I think you've nailed the point about why the ability to down vote is so dangerous and in almost all cases I've heard of leads to bullying not helping quality. Flagging to request real review of content or to add meta data like NSFW (again with actual review) is useful. Flagging as a mechanism for trolling is not. So baseless flagging is IMO as bad as baseless trolling.