Adultery is the most awful non-violent thing that one person can do to another.
If people can seek damages due to other forms of stress, then adultery is certainly warranted as a form of stress.
But as far as the secular government is concerned, marriage is a contract. Unless there is a pre-nup in place, I'm not sure that punitive damages are applicable.