The numbers of Nazis, WNs, and other extreme groups lumped in with the right wing are so tiny they constitute virtually a zero menace to minorities.
So what the left is attempting to do is maintain that every person and group on the right is extreme and deserves to be denounced or silenced.