I'll never stoop to the level of outright dismissing a person's opinion via my own assertion that it doesn't affect the person who has the opinion.
That said, it's not hard to notice that a lot of the people who will call "transphobia" on people who express concerns about males competing in women's sports are also people who don't care about sports.
It's not even that these people aren't sports fans. These tend to be people who, for the life of them, couldn't understand why I flew across the country to watch a football game last year. It's not just that they value their "social justice" causes more than sports. It's that they literally don't care about sports but for the fact that there's a social justice cause for them to pretend to care about.
If that's you. If you think that me going to Buffalo for a football game last year was dumb, and a complete waste of money, maybe you're not the person who should be having this fight about fairness in competition in sports. If that's you, it's probably impossible for you to know what it feels like to devote enough time and effort to a sport to become among the best, just to lose your spot because of a loophole or an unfair rule change.