Poll workers DO NOT HAVE THE AUTHORITY to 'ban' watchers.
I do not understand why the watchers do not simply ignore them.
Is it going to get physical? Anything other than physical force should be ignored.
This is where ambush and escalation forces their hand. Walk in unannounced with watchers and lawyers as backup, with the State on speed dial. They will respond with threats and maybe call the local po po.
You then trump that with state officials and arrest any of them that resist, including and especially the po po. Man the place with a state controlled security force and scrutinize everything that is done till the last vote is counted.