Yeah....that scene with Ray's brother involving his lady friend's son asking to be lifted out of the bathtub was a little unsettling.
I don't understand why he was unable to express the reason to her. 
To express his reasons to her, he would have to expose his having been molested. In effect, in his mind he'd be telling her that he's not a "man"
per se.
That's the at large internal struggle for victims of pederasty.
Having lived the majority of his life dealing with his molestation, and having the idea that contact between adults and children is evil and destructive (specifically adult males and male children), plus the fact that (from understanding the Mickey Donovan character, and imagining the sort of father that he was to his young sons), Bunchy can't process the idea that the sort of contact that Clifford and he were having was not sexual in nature, because of his previous experience under similar circumstances (being a naked child held by an adult).
I think that Bunchy is simply a sexual anorexic who is afraid that having been molested himself by a man that everyone else in his whole world called a "good man" and a man of God, his being a good man doesn't mean that he won't try an molest Clifford.
His lousy explanation, his impotence and his child-like personality are manifestations of a tightly restrained, deeply buried id.