The US has an increasingly "suburbanized population."
People hold a reasonable expectation for their pursuit of "life, liberty, and the pursuit of happiness.
They form government jurisdictions, to maintain public safety, like clean water, measures to mitigate against the spread of contageous diseases.
Individual and group freedoms (rights) extend only to the boundary where the exercize thereof infringes on another's rights.
Early laws dealt with resources like land, grazing, forestry, hunting, water, navigation, air safety, etc.
If this church has a managing authority, they likely purchase liability insurance. That authority should direct the pastor, because his action is criminally negligent. In fact the congregation, should want the guy removed. He could wipe out their financial resources.
Analogy: Owner/manager of public space has knowledge of mold damage.
Said owner takes no steps to remediate said potentially harmful condition. Commercial or residential property. Like apartment dwellers.
Nothing about infringing on religious practice. It is verything about reasonable expectations in modern society.