The thing is, both of our infantile concepts of the far right and the far left have authoritarian, collectivist tendencies.
Yes, the out and proud white supremacists are generally associated with the far right. At the same time, the dude with a swastika tattoo is demonstersbly rare regardless of what people on MSNBC parrot every day.
What's more common is the person on the illiberal left who only stands up for minorities so long as they're useful. These are the people who would punch you if you failed to say that Caitlin Jenner was stunning and brave seven years ago, and turned on her the moment she came out as a conservative.
I've heard far more racism from the illiberal left in reference to people like Thomas Sowell, Walter Williams, and Coleman Huges than I've heard from the far right in regard to anyone. The illiberal left has a concept about how a proper black person, or Muslim, or gay person, or trans-person is supposed to think. The moment any of these people show a glimmer of independent thought, they cast them out like lepers.