One thing that every member of the "but brigade," the people who believe in free speech "but," get right is that most people who demand free speech are doing so out of expedience rather than moral principle. Anybody of any philosophy becomes a free speech advocate the moment that they're he or she is censored. Aryeh Neier talked about the fact that the head of the National Socialist Party of North America had his number on speed dial.
Yes, Nazis support free speech when it's Nazi speech that people are trying to suppress.
This is a constant pain in the ass for real free speech advocates. Since most people only view free speech as an expedient, and know that they'll turn around and stifle the speech of their opposition the moment they gain power, they assume the same of everyone.
Free speech advocates who truly stand on principle must make a conscious effort to make our positions known, and spend most of our time and passion in regard to free speech defending the people who we least want to defend.