I believe you hit on the ultimate test: Will they throw out a black fan saying that to a black player? If not, then I see a class action lawsuit that can certainly be won.
The ultimate test will be throwing a black fan out for calling someone a "cracker". (It won't happen)